►
From YouTube: Stateful Apps On Kubernetes - Bring Them On!
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Stateful Apps On Kubernetes - Bring Them On! - Diane Patton & Gunna Marripudi, NetApp; Scott Surovich, HSBC Bank, USA; Scott Miller, DreamWorks Animation; Lisa-Marie Namphy, Cockroach Labs
Speakers: Gunna Marripudi, Scott Surovich, Lisa-Marie Namphy, Tina Gill, Scott Miller
Worried about brining stateful applications onto your Kubernetes clusters? Learn from the Kubernetes experts at DreamWorks, HSBC and NetApp how you can build and operate at scale. In this session, panel members from DreamWorks, HSBC and NetApp will share their experiences from running data rich stateful applications on Kubernetes for years. The panel discussion will cover the following topics and others as related to data management and protection in Kubernetes. (1) Addressing challenges in managing ‘state’ in stateful application. (2) Use of different tools and capabilities Kubernetes ecosystem has delivered. (3) Best practices to optimize storage resource utilization. (4) Future needs of their expanding Kubernetes environments in hybrid-cloud/multi-cloud.
A
I've
done
this
talk
that
there
really
isn't
any
such
thing
as
a
stateless
application,
like
maybe
the
only
stateless
application
that
exists
is
Hello
World,
possibly
so
everything
has
some
form
of
state
and
so
we're
going
to
kind
of
dive
into
that
a
little
bit
deeper.
So
if
you're
wondering
why
you're
here
wondering
if
you
can
run
stateful
applications
in
kubernetes
I
know
you
all
know
you
can,
but
we
have
some
experts
up
here
who
have
taken
it
to
another
level.
So
why
don't?
We
have
everybody
introduce
themselves,
starting
with
gonna.
B
C
C
A
Funny,
okay,
I
also
just
have
to
do
one
more
thing
to
embarrass
I
love
to
embarrass
the
panelists,
but
in
case
you
haven't
downloaded
this
bought
this
or
read
this
Scott
has
written
a
book.
You
can't
see
what
I'm
showing
them
you
might
be
able
to.
If
you
can
see
my
screen,
yeah
I
had
to
get
this
book
it's
available
on
Amazon
and
he's
an
expert
we
had
to
validate.
We
have
at
least
one
expert
on
the
panel
I'm,
sorry
they're,
all
experts,
okay,
but
back
to
us.
A
D
Sure
so
our
container
Journey
started
in
about
2015.
We
had
a
relatively
unique
opportunity
to
re-implement
the
production
pipeline
for
feature
film,
so
a
pipeline's,
a
collection
of
digital
content
creation
applications,
data
sets
workflows,
how
you
hand
that
stuff
all
around
and
we
decided
the
new
hotness,
was
Cloud
native.
So
we
wanted
to
re-implement
using
microservices,
including
the
databases
that
were
being
used
to
support
these
applications.
D
So
we
took
a
microservices
approach
to
databases
and
wanted
to
deploy
those
in
containers
in
kubernetes
clusters,
as
well
as
the
application
front-end
and
services,
and
the
Journey's
been
great.
Ever
since
we've
gone
from
pre-docker
from
La
lxc
containers
and
our
own
orchestrator
that
we
hand
wrote
I,
don't
recommend
that
moved
into
openshift
and
then
Upstream
kubernetes
is
our
current
deployment
and
then
a
little
bit
of
azure
kubernetes
service.
When
we
need
cloud.
C
Well,
first
of
all,
I'm
a
bank
doesn't
sound
as
cool
as
when
he
said
animated
film,
all
the
pipelines,
but
without
us
he
couldn't
do
films.
So
you
need
money
right.
You
couldn't
pay
for
them,
that's
true!
So
we
started
our
journey
about
the
same
time
around
2015
2016.,
actually
a
little
bit
before
with
mesosphere.
So
if
anybody
in
here
has
been
around
that
and
that
kind
of
went
for
a
couple
years
and
Beverly
took
off
for
us
other
than
Jenkins,
so
we
decided
to
get
over
into.
C
A
So
a
couple
questions
for
you
who
here
we
want
to
know
where,
on
your
kubernetes
journey
you
are
in
so
who
here
is
running
kubernetes
show
of
hands,
keep
them
up.
If
you're
running
kubernetes
in
production,
more
hands
went
up
than
were
before.
That
was
interesting.
Okay,
keep
them
up
if
you're
running
State,
full
applications
on
kubernetes
and
production;
okay,
okay,
all
right!
Well,
that's
actually
more
than
I
would
have
thought
we
did.
I
just
saw
Eric
cruise
through
here,
Eric
and
I.
A
Did
this
talk
about
two
years
ago
in
Barcelona,
a
multi-cloud
version
of
this
talk
and
asked
those
similar
questions
and
they're
a
fraction
of
the
hands
that
just
went
up
just
two
two
and
a
half
years
ago,
cncf
kubecon
Barcelona.
So
we
also
want
to
know
who
you
are
strategists
versus
Implement
implementers
and
actually
what
we
really
mean
like
kubernetes
admin
versus
an
infra
admin
and
by
infra
admin.
A
Okay,
thank
you
for
that
super
interesting
and
I
know
some
of
those
roles
overlap,
especially
in
some
of
the
smaller
companies.
I
know
that
some
of
you
wear
all
the
hats.
We
all
wear
all
the
hats,
so
I
want
to
talk
a
little
bit
about
what
applications
you're
running.
So
I
introduced
myself
as
a
cncf
Ambassador,
but
by
day
I'm,
also
a
head
of
devrel
at
cockwitch
Labs.
So
I.
We
just
published
a
report,
not
cockroach,
but
the
dok
community,
one
of
the
ones
I'm
involved
in
a
kubernetes
report.
A
You
can
download
it
it's
available
as
of
last
week,
really
interesting
information
about
who's,
doing
what
on
kubernetes
who's,
doing
it,
how
who's
making
money
with
it,
how
long
they've
been
in
it.
But
one
of
the
questions
was
what
applications
are
you
running
on
kubernetes
and
the
number
one
answer
was
databases
so
by
day,
I
work
for
cockwitch
labs,
yes,
cockroaches
exist
during
the
day,
and
so
that
is
my
day
job
and
so
I
was
really
happy
to
see.
A
C
Okay,
so,
as
Lisa
was
saying,
what
do
you
run
like
what
kind
of
apps,
in
our
case
we're
a
bank
right
so
payments,
it's
pretty
important
to
a
bank
right
payments,
aren't
flowing
HSBC,
there's
trillions
of
dollars
a
day
that
flow
through
this.
So
our
payment
system
is
on
kubernetes,
it
takes
to
be
migrated
and
it
requires
storage,
because
it's
running
Kafka.
C
So
if
anybody
in
here
has
run
that
you
need
to
put
those
log
files
somewhere,
if
you
put
them
on
the
overlay
file
system,
you're
going
to
get
80
and
evictions,
so
that's
just
one,
the
other
app
we
have
that's
pretty
popular
to
do
this
with
is
IBM
mq,
and
this
one
gets
more
interesting.
You
have
one
container
in
we'll
say
one
a
cluster
another
cluster.
We
replicate
that
data
from
the
prod
to
the
Dr,
so
replication
of
data
is
not
just
a
Heritage
thing.
It's
actually
something
you
do
today.
C
By
doing
that,
today,
application
we
have
saved
72
cores
of
cost
just
for
one
instance
of
that
business.
That's
my
trading
platform,
so
I
like
to
tell
people
those
are
both
two
or
zero
apps.
If
we're
not
trading
and
we're
not
moving
money,
DreamWorks
gets
no
movies
or
something
similar.
We
have
to
have
that
so.
D
Yeah,
it's
amazing
how
that
all
works,
so
the
application
stack
that
we
run
it's
a
collection
of
microservices
that
implement
the
asset
management,
workflow
management
and
data
management
features
of
our
film
Pipeline
and
each
app
there's
about
50
micro
services
that
make
up
a
collection
of
apps
and
each
app's
persists
at
state
in
a
database.
So
we
made
a
Cloud
native
microservices
like
decision
seven
years
ago,
where
databases
should
be
Deployable
just
like
any
other
kubernetes
app.
D
So
we
deploy
a
minimum
three
node
database
cluster
for
every
app,
so
we've
got
about
1500
stateful
sets
running
around
a
little
over
550
database
clusters
Deployable
by
the
developers.
So
if
developer
needs
a
database,
they
say
I'd
like
couch
base,
so
I'd
like
Kafka
or
I'd
like
elasticsearch,
whatever
it
is,
they
need
they
can
deploy
it
using
the
same
CI
CD
tool
chain,
the
same
mechanisms
and
apis
they're
used
to
for
deploying
their
application
itself.
So
the
interesting
thing
there
is,
we
not
only
use
databases
to
persist,
State
for
the
applications.
D
B
So
if
you
can
add
like
one
more
to
some
of
the
apps
that
Scotty
and
Scott
are
talking
about,
one
privilege
of
me
are
like
you
know,
working
at
NetApp
talking
to
a
lot
of
end
users,
people
are
actually
using,
not
only
databases
they're.
Also
using
like
you
know,
Jenkins
comes
across
multiple
times.
People
are
actually
running,
like
you
know,
like
thousands
of
instances,
you
know
one
of
some
other
customers.
We
work
with
they're
running,
like
I
call
like
19
or
clusters.
B
A
Yeah
and
so
we're
not
trying
to
tell
you
what
the
definition
of
a
stateful
application
on
kubernetes
is.
We
really
want
you
to
understand
the
implication
and,
what's
the
implication
of
not
doing
this,
I
mean
and
first
of
all,
there's
not
you're
not
going
to
put
everything
in
kubernetes.
So
let's
start
there,
what
aren't
you
going
to
put
what
what
don't
you
want
to
run
in
containers.
A
D
Apps,
the
the
intent
is
that's
kind
of
talked
about
this
earlier.
We've
got
a
deployment
strategy
if
somebody
needs
to
run
a
thing,
it's
container
first
VM,
second
bare
metal
hardly
ever,
and
the
only
exception
of
the
bare
mineral
rule
is
the
HPC
environment
and
everything
else
is
a
container
first
and
then
a
VM
if
it
has
to
be.
C
A
C
Of
course,
when
the
question
comes
up,
containerization
I'll
use
the
line.
If
I
could
I
would
containerize
my
dog,
so
it's
like
anything,
just
put
it
in
there,
especially
Linux.
Let's
be
honest,
so
you
go
to
Windows,
it's
a
different
story
right.
You
can
do
a
lot
with
windows.net
IIs,
but
in
my
opinion
you
can
move
that
monolith
over
microservices
great,
of
course
right,
but
that
monolith
mq
good
example.
That's
not
exactly
a
microservice,
but
it's
saving
us
a
ton
of
money
in
CPUs.
C
Now
those
CPUs,
because
it's
not
on
a
VM
anymore,
means
no
more
patching.
I,
don't
have
all
these
agents
running
I'm,
not
paying
for
operating
system
licensing
and,
of
course,
overall
containers
will
use
less
resources.
I,
don't
think
I
know
of
an
instance
where
a
container
takes
more
than
that
app
on
an
actual
VM.
B
E
A
Okay,
so
we
for
the
most
part,
have
known
each
other
for
a
long
time
and
in
my
opinion,
you've
been
early.
Adopters
Scotty
I
mean
like
really
really
a
doctor,
probably
Scott
as
well,
but
I
haven't
known
you
as
long
but
the
so
I
in
general
that
you're
ahead
of
most
of
the
folks
that
I
talk
to
most
of
the
folks
that
in
the
user
group
that
I
run
in
the
Bay
Area
San
Francisco
Bay
Area.
A
D
It's
not
a
journey,
really,
it's
a
lifestyle.
You
have
to
decide.
This
is
how
you
want
to
deploy
applications,
we're
all
the
way
through
kind
of
the
first
generation
of
deployment
and
we're
we're
doing
some
refactoring
one
of
the
things
that
you
get
when
anyone
can
deploy
a
database
with
kubernetes.
D
You
have
to
patch
for
you
at
the
track,
we're
at
the
point
now,
where
we're
still
trying
to
hire
people
who
know
how
to
work
kubernetes
and
understand
the
data
model
and
understand
how
to
write
operators
and
understand
how
to
manage
an
ecosystem
like
this.
That's
probably
one
of
our
biggest
challenges,
so,
even
after
seven
plus
years
of
doing
this,
finding
enough
people
who
want
to
support,
develop
and
enhance
an
environment.
That's
running
databases,
as
kubernetes
items
is
pretty
challenging.
Still.
C
D
C
Us
we've,
like
I,
said
it's
been
about
five
six
years,
we're
pretty
far
along
that
Journey
we're
multi-tenant,
already
yeah
I
always
have
been
four
years
ago
at
coupon
people
said
I
was
crazy.
You
know
Mark
heard
that
multi-tenant.
Why
would
you
do
that?
Exactly
like
you
said
this
guy
from
day.
One
cluster
control
are
you,
you
don't
want
control
planes
going
all
over
the
place,
just
better
resource
utilization.
Now
our
problem
is
finding
people
and
then
you're
working
in
an
environment,
that's
highly
regulated.
That
is
not
fun.
C
If
anybody
in
here
works
for
government
and
or
a
financial
institution,
it
can
be
like
molasses,
but
luckily
for
kubernetes,
we've
been
going
really
fast.
So
it's
a
very
rare
occasion
and
I
do
have
some
problems.
We
still
need
education
with
Developers,
so
we
still
have
a
lot
of
developers
that
need
help
and
that's
understandable
right.
They
can
code
code,
the
heck
out
of
java,
but
they
don't
know
how
to
make
a
deployment
or
they
don't
add
to
a
PVC
or
they
create
an
Ingress
rule
from
which,
of
course,
won't
work.
C
A
I
think
a
lot
of
people
here
would
hire
you
another
thing
that
came
out
of
the
the
kubernetes
report.
They
were
asked
what
you
know
what
problems
people
are
having
number
one
security
number
two
was
Talent:
hiring
the
talent,
the
right
Talent,
who
can
write
the
operators
that
are
necessary,
not
enough
Talent
out
there
and
interesting
stat
60
of
the
Enterprises.
These
500
Enterprises
that
were
queried
were
saying
that
we're
hiring
this
talent
in
the
next
three
to
six
months.
A
A
D
We
tried
to
do
a
lot
of
training
internally
and
since
one
of
our
big
stateful
application
pieces
is
the
databases
we
started
with
our
dbas
and
tried
to
teach
them,
you
need
to
administer
a
database.
What
you
also
need
to
understand
its
runtime.
You
need
to
be
enough
of
an
SRE
to
understand
the
kubernetes
environment.
You
need
to
be
enough
of
a
observability,
analytic
type
person
to
understand
the
behaviors
of
your
application
in
this
different
environment.
D
More
importantly,
to
understand
failure
domains
because
the
way
kubernetes
nodes
and
kubernetes
pods
can
fail
is
different
than
like
the
way
a
VM
or
a
physical
machine
might
fail.
So
we
tried
to
train
from
within
where
we
could
I
wish.
There
were
University
curriculum
people
here
who
could
start
to
teach
something
other
than
just
how
to
code
in
Java,
because
that's
not
all
there
is
you
actually
have
to
run
the
code.
C
Yeah
for
us
I
actually
went
internal
for
a
lot
because
being
blunt,
it's
easier
to
learn.
Kubernetes
than
HSBC
policies,
because
again
highly
regulated,
we're
in
63
countries
and
my
team
is
global.
So
they
have
to
deal
with
all
of
those
regulations
and
what
we'll
do
is
the
same
kind
of
training
and
there's
a
pretty
good
book
that
you
know
somebody
told
me
about,
but
aside
from
that,
we
do
working
groups
inside
the
bank
so
we're
doing
it
all
different
hours,
so
Hong
Kong
can
get
on.
C
B
That's
just
Scott
I
think
I
I
these
days.
You
know
there
are
so
many
resources
like
including
your
book.
Like
you
know,
there
are
good
training
programs
and,
more
importantly,
the
skill
set
of
organizations
is
actually
growing
up
like
the
kubernetes
skill
set.
So
we
have
a
couple
of
SRE
teams.
You
know
that
actually
run
our
fully
managed
Services
we're
all
self-host
and.
E
C
D
A
B
Yeah
I
think
you
know.
One
of
the
things
that
we
have
been
talking
about
is
once
you
put
these
stateful
applications
in
production.
You
know,
given
that
most
of
you
guys
actually
raise
your
hand
when
you
did
kubernetes
into
production.
When
you
bring
stateful
apps.
You
know
it
introduces
some
challenges
like
one
of
the
things
that
I
see
talking
to
customers
is
kubernetes
is
on
like
a
nice
kid
and
say
every
four
months.
You
know
you
have
a
new
version
coming
right
so
because
of
the
changes,
and
sometimes
you
know
pretty
useful
functionality.
B
Unlike
you
know,
the
Heritage
apps
that
you
know
Scotty
was
talking
about
the
the
upgrade
Cycles
are
not
like
six
months
nine
months
a
year
like
you
know,
people
really
want
to
upgrade
in
nice
Cadence,
you
know
keeping
in
keeping
a
pace.
So
the
big
thing
that
comes
is
like
you
know.
Now.
If
you're
running
stateful
apps,
like
some
the
db2
I,
mean
some
of
the
databases,
applications
or
Kafka,
or
you
know,
Jenkins
Etc,
you
really
need
to
think
about.
Like
you
know,
yeah
I'm,
upgrading
it
you
know,
but
am
I
causing
any
disruption.
B
So,
for
example
like
when
you
talk
to
some
of
the
customers,
the
disruption
is
massive.
There
may
be,
like
you
know,
18
000
developers,
depending
on
acacd
Pipeline,
so
you
just
can't
afford
to
bring
it
down
right
and
you're.
Thinking
about
like
a
database
instance
like
a
scardia
Scott
was
talking
about,
is
his
billing,
like
you
know,
I
actually
come
across
some
billing
users.
Right
I
mean
he's
talking
about
payment
system,
I'm
talking
about
some
billing
customers
that
app,
you
know,
that's
your
Revenue
Source.
You
can't
bring
it
down.
B
B
C
I'll,
add
to
that,
is
it
can't
be
your
source
of
Truth?
So
what
I
mean?
Is
you
get
your
clusters?
You
better
have
backup
right,
A
lot
of
people
say
kubernetes
doesn't
need
the
backup
well
that
data
still
sits
somewhere
right,
I,
don't
care
if
it's
nvmes,
if
it
happens
to
be
in
a
filer,
you
know
solid
fire
EMC,
it
doesn't
matter
right.
You
still
need
something
to
back
that
up
so
also
know.
Your
storage
know
your
storage
classes.
What
you
can
do
with
these
storage
classes,
you
have
to
understand,
read,
write
once
read.
C
C
E
That
pretty
much
takes
away
the
whole
point
of
kubernetes,
also
because
the
way
to
get
around
that
or
one
way
to
get
around
that
would
be
to
use
to
recreate
right.
And
then
you
don't
have
your
rolling
upgrade
unless
you
want
to
start
working
with
affinities
and
things
like
that
which
make
everything
a
lot
more
complex.
D
Yeah,
the
other
thing
to
keep
in
mind
is
where
the
data
might
live.
We
made
a
conscious
choice
to
put
most
of
our
database
durable
data
in
the
cluster,
distributed
amongst
the
worker
nodes,
using
various
virtualization
techniques
to
present
back
a
PV
that
makes
upgrades
of
the
storage
environment
and
the
kubernetes
environment,
and
your
application
is
all
dependent
on
each
other.
So
it's
doable
it's
a
good
choice
for
databases,
because
you
get
really
good,
read,
write,
latencies
and
really
consistent.
D
I
o
without
Jitter,
but
the
downside
is
you've
created
another
dependency
in
the
kubernetes
cluster,
which
you
can't
just
reboot
a
node,
because
it's
misbehaving
you
have
to
think
about
what
other
data
services
might
be
affected.
If
you
put
your
data
outside,
say
of
some
sort
of
a
storage
array
for
the
durable
storage,
then
you
have
to
protect
it
using
its
native
mechanisms.
So
you
have
what
you
do
in
kubernetes
to
deploy
and
operate.
A
Okay,
I've
got
maybe
two
more
questions
for
you,
then
we'll
open
it
up
to
the
room.
Kubernetes
first
comes
out.
People
think
it's
like
it's
going
to
be
everybody's
savior.
Let's
just
do
everything
with
it.
It
can
do
all
the
things
we
found
out
it
couldn't
do
all
the
things.
What
were
some
of
the
you
talked
about
where
it
went
right?
What
are
some
of
the
places
where
it
went
wrong
because
that's
always
more
interesting
to
hear
right.
D
Golly,
where
did
it
go
wrong?
I
I
think
that
to
me,
one
of
the
interesting
early
on
things
is
the
fundamental
behavior
of
kubernetes.
If
a
pod
doesn't
run,
it
just
starts
it
again
and
again
and
again
and
again
until
it
finds
some
either.
You
know
you
get
tired
of
it
or
it
finally
ends
up
running.
You
can't
do
that
with
a
database
node,
and
you
can't
do
that
with
a
storage
node.
You
have
to
think
consciously
about
where
it
resides.
D
C
B
Speak
some
more
and
you
can
even
so
some
of
the
challenges
you
know
these
guys
have
been
doing
it
for
a
long
time.
You
know
I
work
with
some
of
the
customers
who
are
kind
of
maybe
here
into
it,
and
maybe
two
years
like
relatively
speaking,
early
in
their
life
and
the
maturity
I'll
talk
from
a
data
perspective.
You
know
there
may
be
other
issues.
You
know
that
you
guys
can
chip
in
you
know
how
you
provision.
Storage,
like
you
know,
static
touch
like
you
know,
some
very
key
Concepts.
B
You
know
very
where
data
is
residing
where
your
storage
is
coming
from,
but
the
other
thing
is
also
how
we
are
provisioning,
that
storage
volume
to
your
part,
the
persistent
volume,
how
is
storage
volume,
gets
mapped
to
a
persistent
volume?
Some
other
you
know,
I,
don't
got
a
mistake,
you
know
if
you
guys
are
using
don't
get
offended,
is
now
people
actually
use
like
the
traditional
way.
You
know
you
call
your
storage,
it
may
ask
him
or
heard
to
create
a
storage
volume
map
it
to
a
part.
B
That's
called
Static
provisioning
right
like
right
now,
like
you
know,
every
storage
array
vendor,
like
you
know,
if
you
guys,
are
running
in
the
cloud
pretty
much
every
cloud
provider
is
providing
a
CSI
CSA
driver
that
gives
you
Dynamic
provisioning
and
some
other
thing
that
Scott
was
talking
about
it.
Lets
you
provision
different
kinds
of
volumes
to
match
your
quality
of
service
requirements
or
availability
requirements.
So
there
is
a
lot
of
power.
You
know
that
you
don't
need
to
make
the
same
mistake
like
you
know.
B
E
And
by
using
some
of
the
CSI
drivers
also,
you
can
also
hand
some
of
this
off
to
the
developers
themselves.
So
this
way
you
can
basically
maybe
create
some
application
patterns
where
the
developers
can
then
say
Hey,
you
know,
I
would
like
a
certain
type
of
storage
for
my
application,
a
certain
amount
of
storage.
Then
they
just
need
to
click
that
through
Automation
and
then
they
can
get
what
they
need
in
order
to
deploy
their
application
as
well.
C
Yeah,
actually
that's
one
thing
I
do
like
is
going
back
to
my
IBM
mq.
The
developer
can
actually
cut
that
disk
over
themselves.
Unlike
the
old
days
of
like
EMC,
you
might
call
your
storage
team
and
say
split
this,
and
then
they
go
and
do
it.
The
developer
can
actually
just
run
a
yaml
and
it
cuts
it
over
and
when
they're
done
it
can
sync
back
both
async
or
sync,
depending
what
you
really
need.
E
A
Were
talking
earlier,
actually,
Scotty
you've
been
going
to
a
lot
of
sessions
here
and
shout
out
to
the
community
there's
actually
a
lot
of
work
that
has
been
done
and
you
were
talking
about
in
the
last
session.
Some
of
the
things
that
are
coming
out
with
kubernetes
are
solving
a
lot
of
the
problems
that
we
used
to
have
to
try
to
solve
manually
yeah.
D
In
the
session
after
this
one's
about
SQL
and
kubernetes,
so
if
you're
interested
stay
the,
but
some
of
the
other
things
I
found
out
about
is
like
the
ability
to
treat
a
database
as
a
as
a
service
connected
back
into
kubernetes.
Some
of
the
things
coming
up
make
some
of
the
issues
that
we
both
ran
into
over
the
years
kind
of
non-starters
the
switch
to
CSI
as
the
way
to
both
dynamically
and
statically
provision.
Your
storage
is
interesting.
A
B
Yeah
so
again,
I'll
go
back
to
my
the
data
perspective,
so
please
add
the
rest
of
it
like
from
data
perspective.
You
know.
One
of
the
things
we
see
is
Scott
kind
of
mentioned.
Like
you
know,
in
his
thing,
is
there
are
the
the
so-called
Heritage
apps
also
have
best
practices?
There
are
certain
things
that
actually
keep
this
Mission
critical
applications
running
right.
He
talked
about
no,
don't
think
backup
as
like
you
know,
a
second
thought,
but
not
only
just
backup.
B
You
know
business
continuity,
like
you
know
some
other
customer,
especially
that
when
you
try
to
do
if
you're
already
in
the
cloud,
you
really
need
to
think
about.
Oh
what
about
cross
region
failures
if
you're
running
on-prem,
you
know
you
really
need
to
think
about,
or
can
I
replicate
that
kind
of
infrastructure
in
my
clouds?
So
really
the
replication.
You
know
Scout
reselling
is
very
critical,
like
you
know,
synchronous
asynchronous,
so
that's
keep
in
mind.
B
C
One
thing
I'd,
like
typically
see
get
added,
is
more
Mount
options
for
a
PVC,
so
we
do
a
lot
of
things
externally
as
well,
so
you
might
go
to
an
NFS
mount
that
Windows
uses
or
Mainframe
uses
and
then
kubernetes
don't
use
for
kubernetes
to
get
there.
You
put
every
node
in
the
export
list.
That
means
any
pod
can
try
to
get
that.
There's
nothing
built
in
the
kubernetes.
That
can
stop
this.
C
A
D
The
future
of
you,
one
of
the
things
that
we
struggled
with
since
we
did
choose
to
offer
10
different
database
types
to
developers
deployed
as
in
kubernetes
how
you
get
credentials
passed
back
to
the
application.
How
do
you
get
a
standard
way
to
do
a
quest
backup?
We
need
kind
of
all
the
database
vendors
maybe
to
get
together
and
come
up
with
a
community
standard
for
what
the
operator
looks
like
that
manages
a
database.
What's
your
health
endpoint
looks
like
what
your
status
endpoint
looks
like,
so
that
everyone
reports
the
same
stuff.
D
I
shouldn't
have
to
have
10
different
sets
of
making
sure
a
database
is
healthy
for
10
different
database
types
because
they're
all
fundamentally
databases-
and
there
might
be
some
specialization
I'd,
just
like
to
see
some
agreement
about
everyone,
doing
a
database
deployment
and
kubernetes
the
same
way.
Queen.
A
G
D
Yeah
I
mean
we.
Our
our
model
for
geographic
distribution
is
there'll,
be
two
instances
of
the
database.
The
databases
themselves
will
replicate
with
whatever
their
native
engine
is
we're
looking
for
ways
to
maybe
replicate
using
the
storage
underneath,
especially
when
you
scale
let
the
storage
replicate
and
then
let
the
database
replicate
and
then
it
can
reconcile
itself
still
in
early
discussions.
But
for
us
it's
all
about
getting
a
deployment
in
two
different
clusters
with
a
configuring
database
replication.
So
it's
a
multi-cluster,
behavior
I.
C
Don't
know
a
good
question
had
to
do
with.
Partly
you
got
to
worry
about
your
round
time.
You
know
Network
traffic,
it
should
be
like
10
or
under
roughly
anything,
at
least
for
synchronous.
Right,
async's
gonna
be
a
different
story,
but
you
got
to
make
sure
you
have
that
and
bandwidth
there's
a
lot
of
stuff
going.
D
G
C
G
F
All
the
tools
needed,
so
the
developers
can,
you
know
straight
away,
deploy
these,
but
there's
also
like
a
cost
Associated
from
a
support
angle.
So
to
speak
to
you
know,
because
these
are
not
just
application
Parts.
These
are
actually
databases,
so
you
can't
just
have
them
go
down
all
the
time
and
if.
F
Need
a
you
know,
whole
support
model,
so
I
wanted
to
understand
what
kind
of
challenges
you
have
seen
over
the
years
in
helping
build
that
SRE
team
or
kind
of
solutions
that
you
had
in
that
aspect.
There
was
one
question:
I
had
and
just
one
more,
what
kind
of
tooling
did
you
choose
in
your
entire
solution
in
terms
of
cicd
package
management?
F
D
D
Tooling
and
CI.
Cd
pipelines
is
a
whole
other
couple
sessions,
but
one
of
the
things
we
did
is
we
put
our
databases
in
separate
physical
clusters
from
the
stateless
part
of
the
application
stack
so
that
we
can
treat
those
physical
clusters
differently.
They're
provisioned,
with
different
Hardware
there's
disks
in
the
database
clusters.
We
have
different
operating
procedures
around
them.
Probably
the
biggest
operational
challenge
is
when
debugging
something
the
dbas
have
to
put
on
their
DBA
hat
and
their
SRE
hat
to
understand.
If
it's
the
database
is
it
kubernetes?
Is
it
the
network?
D
A
Good
question:
we
actually
had
that
question
on
here,
but
I
didn't
have
time
to
ask
it
so
I'm
glad
you
did
so.
That
was
the
last
question.
Oh
he's,
shutting
us
down
all
the
other
sessions
have
gone
way
over.
It's
okay,
we'll
be
here.
If
you
have
any
more
questions,
but
I
really
want
to
first
of
all
thank
Ghana
for
putting
a
lot
of
this
content
together
and
for
getting
a
lot
of
this
panel
organized
and
together.