►
Description
Join Rasheed Amir, Kubernetes evangelist and CEO at Stakater, as he discusses how companies are taking advantage of Kubernetes when it comes to DevOps for their mission-critical cloud-native applications. These applications however will have business requirements for near 100% uptime and availability. In the event of a disaster, the applications need to continue to be available and perform without a hitch. For DevOps engineers with non-containerized as well as containerized architecture background, it is important to understand the components and strategies they need to consider for effective disaster recovery for Kubernetes.
You will learn:
- Some concepts and terms to consider for disaster recovery business needs
- Kubernetes architecture for ensuring fault tolerance and high availability
- Factors to consider while creating a Disaster recovery plan
- The components for which to implement backup and restore
Presenter:
Rasheed Amir, CEO @Stakater AB
A
Okay,
let's
get
started
welcome
everyone.
Thank
you
very
much
for
joining
us
today.
Welcome
to
today's
cncf
webinar,
effective
disaster
recovery
strategies
for
kubernetes,
I'm
jerry
fallon,
and
I
will
be
moderating
today's
webinar
I'd
like
to
welcome
our
presenter
today,
rasheed
amir
ceo
at
socketeer,
a
b
just
a
few
moments
before
we
get
started
during
the
webinar.
You
are
not
able
to
talk
as
an
attendee.
There
is
a
q,
a
box
at
the
bottom
of
your
screen,
so
please
feel
free
to
drop
your
questions
in
there
and
we'll
get
to
as
many
as
we
can.
A
B
B
So
as
together
we
have
been
lucky
to
work
with
kubernetes
since
2015
and
we
call
ourselves
been
looking
to
lucky
in
a
sense
that
it's
really
an
awesome.
I
would
say
technology
that
we
got
a
chance
to
start
working
on
and
since
then
we
have
been
working
on
different,
I
would
say
projects
applications
assignment
with
customers,
and
we
have
learned
a
lot
about
how
how
you
can
actually
have
a
disaster
recovery
for
strategies,
and
we
have
learned
the
hard
way
we
have
failed
and
then
try
to
figure
out
okay.
B
B
So
staccato
is
basically
just
a
quick
like
we
are
cncr
certified
solution
provider
for
kubernetes.
We
are
a
kubernetes
enablement
company
and
helping
companies
to
realize
the
full
potential
of
kubernetes
it's
in
us,
along
with
its
ecosystem,
for
starting
from
the
strategy
to
development
operations
and
help
in
all
different
dimensions
within
an
organization
to
take
how
they
can
take
full
complete
benefit
from
kubernetes.
B
I
don't
think
I
need
to
talk
about
like
kubernetes,
as
I
think
most
people
who
are
here
are
must
be
using
kubernetes.
I
don't
need
to
say
like
definitely.
Kubernetes
is
the
leading
container
orchestration
platform
and
given
its
beauty
of
extensibility,
it's
it's
really
too
awesome
to
to
work
with
so
talking
about,
like
today's
challenges
for
high
availability
and
like
disaster
recovery.
So
what
are
the
challenges
for
today?
B
If
you
look
in
the
past,
it's
completed
class
now
things
have
changed
quite
a
lot
with
respect
to
the
high
availability,
availability
and
disaster
recovery,
and
if
you
look
at
like
today's
requirement
from
cios,
they
want
zero
rpgs.
They
would
like
to
react
quickly
to
demand.
They
cannot
like
to
scale
out
and
of
course
they
want
24,
7
and
365
availability,
which
turn
means
that
how
to
have
that
kind
of
requirements
you
for
implementation.
You
need
to
have
the
things
like
devops.
B
So
the
question
is
what
is:
is:
is
business
continuity
as
a
business.
Continuance,
of
course,
is
all
about
maintaining
the
critical
business
functions
during
and
after
disaster
has
occurred
and
the
business
continues
defines
two
main
criteria:
recovery,
point
objective
and
the
recovery
time
objective.
B
So
a
disaster
recovery
can
be
viewed
as
the
implementation
of
rpo
and
rto,
and
I
will
show
later
on,
like
what
exactly
rp
and
rto
are
when
a
disaster
happens.
So
business
continuity
is
is
basically
first
you
have
a
backup
and
then,
when
it
happens,
you
are
able
to
recover
from
that
back
and
then,
based
on
what
rp
and
rtls
you
had.
B
And
then
why
is
business
continuity
important?
And
so,
when
your
business
depends
on
that
time
of
your
services
and
applications,
it
is
critical
to
have
a
comprehensive
data
protection
disaster
recovery
strategy.
So,
regardless
of
whether
the
workloads
are
running
on
premises
in
the
cloud,
disruptions
can
cost
your
business
thousands
of
dollars
each
minute
in
lost
revenue
and
decrease
in
worker
productivity
and
damage
to
your
brand
loyalty
and
reputation.
B
So
disruptions
can
take
any
forms
like
user
error,
misconfigurations
or
stuff
like
that,
and
but
the
question
is
okay:
we
want
to
have
a
business
continuity,
but
then
what
are
the
challenges
to
implement
that
kind
of
discontinuity?
So
you
have
to
look
into
things
like
efficiency,
because
dr
requires
a
regular
testing
and
the
event
of
disaster
resources
must
be
available.
B
This
leads
to
sometimes
we
have
either
resources.
99
percent
of
the
time,
then,
of
course,
complexity.
So
updating
applications
in
is
complex
enough,
but
dr
requires
a
complete
redeployment
where
the
dr
side
almost
never
mirrors
production
due
to
cost.
So
it
could
be
complexity.
The
challenge
and,
of
course,
cost
they
are
usually
is
at
least
double
doubles
the
price
so,
depending
on
how
you
are,
are
you
doing
stuff
there
are?
These
are
the
different
challenges
that
you
have
to
keep
in
mind,
along
with
your
rpo
and
rto,
to
figure
out?
B
Second,
one
is
called
the
dr
application
where
you
can
fail
over
to
your
recovery
site,
which
is
also
traditional
architecture
as
well,
that
you
have
two
sides
and
on
failover
you
move
to
the
other
side,
and
then
you
have
this
option.
What
is
called
multi-site
application,
which
is
more
cloud
native,
so
you
deploy
across
multiple
sites.
Disaster
recovery
is
built
in
simply
a
rescale
and,
of
course,
it's
recommended
to
have
three
sides
versus
as
compared
to
having
two
sides.
B
B
B
Is
that
is
a
timeline
and
if
you
take,
if
you're
taking
backups,
let's
say
every
10
hours
or
5
hours,
and
if
the
disaster
happens,
then
you
will
have
a
downtime
until
you
become
green
again
and
then
it
means
that
that
is
your
return,
the
recovery
time
objective
from
the
time
the
disaster
happened.
When
were
you
back
again?
We
want
to
be
back
in
10
minutes,
one
hour
or
two
hours,
six
hours,
that
that
depends.
The
business
has
defined.
B
What
what
like
for
them?
How
much
does
the
downtime
cost
them
and
then
the
second
thing
is
data
loss.
Of
course
there
is
no
like
recovery
point
of
objective
like
how
much
data
loss
can
you
can
you
bear
with
one
hour
five
minutes
ten
minutes,
so
these
are
like
the
different
metrics
that
you
have
to
keep
in
mind
and
then
based
on
these
metrics.
Actually
you
will
define
your
dr
strategy
so
whatever
options
I
will
present
afterwards,
you
have
to
always
keep
in
mind
these
two
values.
B
So
if
you
go
back
to
businesses
and
say
they
say,
we
need
a
dr
strategy,
so
you
would
have
to
start
your
discussion
with
these
two
mattresses,
because
once
the
business,
the
c-level
executives
or
whatever,
whoever
they
are
like
the
cn
makers,
once
they
define
these
metrics,
then
you
can
propose
them.
Okay.
What
kind
of
dr
strategy
you
should
have
that
will
actually
meet
your
these
objectives.
B
Of
course,
now
you
have
solutions
available
when
you
have
live,
replication
is
happening
and
the
data
loss
is
almost
zero
or
like
it's
very
minimal,
you
can
have
almost
zero
downtime.
So
things
are
all
all
the
things
are
possible
right,
but
again
again
that
depends
what
are
the
matrices
that
the
customer
has
defined
for
you
they
want
to
have.
B
So
if
you
talk
about
before
the
container
era,
things
are
much
much
simpler
because
you
had
a
one-to-one
mapping,
relationship
between
application,
servers
and
applications.
So
what
you
really
do-
or
you
could
do-
was
obtain.
Let's
take
a
backup
of
everything
in
my
server.
I
have
a
vm,
let's
take
a
snapshot
and
then,
if
a
failure
happens,
I
will
just
start
a
new
world
new
server
or
new
application
with
the
same
personal
machine
or
snapshot
that
I
have
perfectly
fine.
B
So
that
was
things
were
pretty
pretty
easy,
of
course
before
we
then
into
containers
started
doing
microservices
stuff
like
that.
So
what
are
the
traditional
dr
approaches
that
you
might
have
seen
out
still
being
used
as
well?
So
I
mean
roast
number
one:
is
you
can
have
a
cold
standby
on
the
left
side?
You
have
an
active
site
on
the
right
side,
you
have
a
standby
side
or
you
can
have
an
a
worm
standby.
So
the
difference
between
the
two
is
the
user
application.
B
So
if
you
have
a
cold
standby,
then
you
are
taking
periodic
backups
and
restoring
them,
and
if
you
have
a
warm
standby,
that
means
more
like
a
batch
or
a
continuous
replication
is
happening
so
you're
moving
stuff
from
one
to
the
other
you're
copying
over
stuff
with
some
way
so
application
and
data
are
unavailable
until
the
standby
is
brought
online.
Data
loss
highly
likely
in
this
scenario,
because
you're
either
taking
periodic
backups
or
you're
taking
a
batch
or
maybe
continuous.
B
But
again
there
will
be
some
batching
going
on
the
approach
number
two
that
you
have
is
again
similar
to
that
one.
So
you
have
a
continuous
replication.
In
this
case
you
call
hot
standby
and
then
the
other
option
is
hot
standby,
with
read
replicas.
So
in
this
case,
you
are
have
a
columnist
replication
going
on
and
you
have
a
hot
standby
and
then
you
can
also
have
a
hot
standby,
but
with
read
replicas,
so
the
right
is
only
on
the
active
one,
but
read
can
be
done
on
the
standby.
B
B
In
that
case,
you
will
have,
of
course,
minimal
data
losses
and
stuff
like
that,
because
you're
relying
on
the
underlying
your
vm
solution
or
your
storage
solution
to
actually
help
you
replicate
your
data
and
help.
You
replicate
your
virtual
machines
if,
in
case
the
disaster
happens,
so
these
are
kind
of
the
traditional
approaches.
So
when
we're
going
into
the
container
era,
these
we
will
just.
We
will
borrow
from
these
traditional
approaches
which
are
still
being
used
underlying
in
our.
B
B
So
you
don't
have
to
bother
about
okay
thing,
okay,
run
here
around
there,
it's
like
it
picks
up
and
runs.
Of
course,
you
can
have
node
pools
and
those
things
like
to
say:
okay,
just
run
on
this
kind
of
nodes.
That
is
possible,
but
it's
just
like
there's
no
more
one-to-one
mapping
anymore,
so
you
have
even
more
dynamic
architecture
now
as
compared
to
pre-container
era.
B
So
then
comes
down
to
kubernetes,
backup
and
recovery.
So
why
backup?
Of
course
you
could
have
a
national
or
man-made
disasters.
It
could
be
human
error,
it
could
be
hacker,
thingy
malware
or,
of
course
you
can
have
legal
standards
or
compliance
regulations
that
you
need
to
take
back
up.
So
even
if
you're
using
kubernetes,
you
need
to
think
about
your
backup
and
recovery
strategy.
Your
dr
strategy
and
the
reasons
could
be
a
lot
and
then
the
question
comes
okay.
What
do
I
need
to
backup
when
I'm
running
my
kubernetes?
B
A
B
Related
to
your
applications
that
you
are
deploying
to
your
cluster,
your
operators
or
everything
anything
that's
deploying
you
need
to
have
a
backup
of
that
so
and
then,
of
course,
your
access
controls,
how
people
access
the
cluster
and
stuff
like
that.
If
you
have
our
backs
network
policy
is
defined,
they
need
to
be
backed
up.
Then,
of
course,
another
important
thing
is
certificates
that
are
being
used
for
your
routers
and
stuff,
like
that.
B
That
needs
to
be
properly
backed
up
as
well,
and
then,
of
course,
the
last
key
is:
is
the
persistent
volumes
if
you're
running
stateful
applications
within
your
new
kubernetes
cluster,
so
you
have
to
so
you
have
to
back
up
all
of
this
stuff,
and
if
you
look
into
this
data,
you
can
divide
into
three
things.
Basically,
the
volumes.
B
So,
if
you
look
into
that,
you
can,
we
can
dis.
We
can
divide
our
like
kubernetes,
that
it
has
two
sort
of
components.
One
is
what
we
call
stateful
components
and
the
other
is
what
we
call
stateless
components.
So
when
we
talk
about
stateful
components,
you
have
two
things:
one
is
each
city.
This
is
where
the
kubernetes
cluster
state
is
stored.
It
is
the
brain
of
kubernetes
and
of
course
it
holds
all
the
application
configurations,
and
the
second
thing
that
you
have
is
is
actually
the
persistent
volumes.
B
So
if
you
are
running
stateful
workloads,
then
you
need
to
have
a
persistent.
You
will
have
precision
volumes,
so
you
need
to
think
about
these
two
components.
When
you
are
thinking
about
your
dr
strategy,
how
on
am
I
going
to
keep
a
backup
of
my
ecd
and
how
I'm
going
to
keep
a
backup
of
my
persistent
volumes?
B
So
the
other
two
two
things
that
you
have
to
always
answer
and
then,
of
course
you
have
a
lot
of
stateless
components
which
are
like
rest
of
the
communities,
control,
plane
and
the
worker
node
components
and
the
stateless
workloads.
So
they
are,
you
have
lots
of
them
as
well.
Right
but
again
you
don't
have
to
bother
at
all
about
the
stateless
components
you
already
have
what's
running
on
your
kubernetes
cluster
is
inside
your
ecd,
so
you
don't
have
to
worry
about
like
keeping
it.
B
B
It's
consistent
and
highly
available
stores.
All
the
cluster
state
and
data
all
kubernetes
objects
in
the
cluster.
So
if
you
look
into
kubernetes
what
is
kubernetes
kubernetes
is
a
really
awesome,
well-defined
api.
So
it's
in
a
very
well-defined
rest
api
where
you
can
just
go
and
interact
with
and
okay.
Ask
him.
Okay,
create
me
deployment.
Create
me
a
config
map
delete
this.
Do
this,
so
it's
really
well
defined.
Set
of
rest,
api,
endpoints
or
proper
entities
which
are
being
maintained
over
our
apis.
B
So
if
you're
running
it
in
the
cloud
or
locally,
you
can
just
take
the
volume
snapshot
as
well,
and
the
third
option
is
that
you
backup
the
kubernetes
object
and
resources.
So
that
is
the
third
option
that
you
can
just
take
a
backup
export
all
of
the
apis
objects
and
put
them
out
somewhere
export
and
then
later
or
you
can
import
them
back.
B
So
there
are
like
three
major
options
of
backing
up:
an
hcd
cluster
and,
of
course,
depending
on
how
you
actually
want
to
do
your
recovery,
then
you
can
follow
the
documentation
of
the
distribution
that
you
use,
how
they
suggest
that
you
recover
your
ecd
core.
So
for
different,
I
would
say:
distributions
of
kubernetes
you
have.
They
have
different
documentation
procedure.
How
to
recover.
So
backup
is
one
thing.
Then
recovery.
B
You
have
to
follow
some
manual
steps
to
recover
your
hcd
core
so
that
your
control
plane
comes
back
operational,
so
just
to
put
just
to
explain
like
if
you
haven't.
If
you
have
a
running
cluster,
and
if
your
control
plane
becomes
unresponsive
or
becomes
like
goes
down,
it
will
not
affect
your
running
applications
as
such,
because
they
will
be
still
be
accessible,
but
you
will
not
be
able
to
to
create
new
applications
or
deploy
new
stuff,
and
then
you
have
to
restore
each
form
to
bring
things
back
so
restore
ecd.
B
Csi
volumes
are
not
that
new,
but
yeah
bit
new
as
compared
to
non-csi
volumes
and
if
you're
using
a
csi
volume,
then
creating
a
snapshot
or
persistent
volume
is
just
creating
another
manifest
and
that's
it
then
you
can
create.
You
can
say:
okay
create
a
snapshot
of
my
from
a
volume,
so
it
will
create
a
snapshot
for
you
and
then,
of
course,
you
can
define
your
strategy.
B
How
often
should
it
take
the
snapshot
and
stuff
like
that
and
then,
but
of
course
you
have
to
keep
in
mind,
you
have
to
check
the
csi
driver
that
you
are
using.
Does
the
volume
snapshot
functionality
is
implemented,
I
mean,
does
it
have
really
underneath
a
volume
snapshot
functionality?
If
it
does,
then
csi
based
volume
is
a
really
good
way
like
it's
easy,
easier
to
define,
because
again
you
can
put
your
snapshot
strategy
as
code
as
well.
B
B
C
A
B
And
then,
if
you
have
non-csi
based
volumes,
then
you
have
to
look
at
the
vendor
vendor
like
if
you
have
like
non-csi
based
volumes
which
you're
using
you
know,
then
you
have
to
look
into
vendor
specific
backup
restores
or
you
can
look
into
the
open
source
solutions
that
are
available.
If
you
want
to
take
a
look
to
them,
so
that's
doesn't
mean
if
you
know,
if
you
don't
have
csi
based
volumes,
you
cannot
do
backup
and
recovery
for
volumes.
You
can
still
do
that
and
there
are
some
open
source
solutions.
B
Otherwise
you
can
use
the
underlying
solution
that
you
are
that
are
available
from
the
whatever
technology
you
are
using
in
your
enterprise,
be
it
like
data
storage
technology
or
whatever
technology
using
they
have.
They
have
options
for
backup
and
these
two.
So
if
you're
looking
at
public
clouds,
they
have
built-in
snapshot
capabilities.
You
can
use
those
snapshot
capabilities
as
well
as
your
volumes.
B
So
now
we
have
understood
that
we
have
to
backup
ecd
and
then
we
have
to
back
up
our
volumes.
Now
we
will
talk
about
the
different,
like
restore
your
strategies.
What
are
the
restore
strategies
that
we
have
for
our
for
our
platform
operations?
What
strategies
do
we
know?
Can
we
look?
Can
we
define
into
so
what
are
those
strategies
so
enough?
How
we
have
learned
in
our
experience
that
you
can
define
into
two
kind
of
strategies?
One
is
what
you
can
say
rebuild
like
you're
rebuilding
and
the
other
option
is
repair.
B
So
you
can
have
what
we
call
plaque
platform,
backup,
restore.
You
can
restore
vms
from
a
snapshot.
You
can
fail
over
to
another
cluster.
You
can
fail
over
to
other
site
and
then
you
can
rebuild
from
scratch.
It
ops.
So
I
mean
in
our
experience,
we
have
done
these
kind
of
different
strategies
and
learned
from
them.
B
Each
will
have
kind
of
have
their
own
pros
and
cons,
and
some
are,
of
course,
are
better
than
the
other,
but
again
that
that
all
depends
on
your
rp
and
rtu
and
how
much
money
you
have
to
spend
on
on
your
backup
and
recovery
strategy
like
your
dr
strategy,
so
as
by
the
name.
What
is
the
the
first
strategy
platform?
B
Backup
and
restore
is
if
you're,
on
the
left
side,
you
have
your
source
cluster
on
the
right
side,
you
have
your
target
cluster
and
then
you're
running
some
backup
tool,
which
is
actually
taking
a
backup
of
your
applications
like
hcd,
basically
and
your
persistent
volume.
Then
your
images-
and
it
is
storing
that
information
at
some
in
some.
B
Repository
so
everything
is
being
backed
up
and
put
up
there,
and
then
you
have
your
target
cluster.
Where
you
can
use,
you
can
use
the
same
backup
tool
to
restore
from
that
stuff,
so
you
can
use
some
backup
tool,
for
example,
layer
could
be
the
one
that's
an
open
source
or
you
can
use
others
they're
out
there
as
well,
but
but
back
up
your
cluster
resources
and
volume.
So
then
try
cluster
or
you
can
also
back
up
a
part
of
the
cluster.
B
B
Of
course,
you
will
require
adequate
storage
in
your
replication
repository
where
you're
putting
up
the
stuff
and
when
it's
needed,
you
can
restore
the
entire
cluster
to
its
working
state.
In
a
new
instance,
you
start
a
new
instance
of
your
cd
cloud,
so
you
start
a
new
cluster
and
then
you
restore
everything
from
your
from
your
your
backup,
the
things
that
you
have.
So
that's
like
one
strategy
that
you
can
use
and
it's
used
quite
commonly
as
well
strategy.
B
B
So
if
you
are
taking
a
valium
volume
backup
of
the
ecd
and
then
you
can
restore
the
new
cluster
from
that
from
those
following
snapshots
and
then,
of
course,
add
new
worker
nodes
for
the
volume
backups,
you
can
follow
the
same
strategy.
But
again,
in
that
case,
you
will
have
to
back
up
your
volumes
and
bring
them
back
on
your
new
cluster.
So
you
could
have.
B
The
the
third
strategy
is
basically
is
to
fail
over
to
another
site.
So
what
is
a
failover?
So
in
that,
in
the
case
of
failover
of
one
cluster,
you
use
failover
cluster.
So
both
clusters
are
nearly
identical
infrastructure,
identical,
stateless
application,
identical
configurations
could
be
different,
but
clusters
are
kept
synced
with
a
parallel
ci
cd.
B
If
you're
running
stateful
applications,
then
you
have
to
rely
on
your
underlying
technology
to
to
ensure
that
the
data
is
replicated
from
the
active
site
to
the
standby
site
as
well.
So
in
case
active
goes
out
of
service.
Standby
can
take
it
up
and
if
you
have
stateful
applications
it
should
it
should
see
the
same
volume
or
data
around
up
there.
So
that's
another
strategy.
Again.
If
you
look
at
your
rp
and
rto,
then
this
one
you
look
at
I
mean,
then
this
one
will.
B
Definitely
you
have
to
consider
from
the
cost
perspective
that
it's
like
doubling
your
cost.
You
have
to
run
two
clusters
in
parallel
and
you
have
to
maintain
two
clusters
in
parallel
and
stuff
like
that.
So
that's
like
that's
one
strategy
as
well,
then.
Another
strategy
that
is
very
common
now
is
actually
a
dislike
that
you
usually
see
in
your
public
cloud
setup
that
you
have
one
cluster,
which
is
stretched
over
multiple
sites,
and
you
can
do
same
thing
on
for
on-prem
clusters
as
well.
B
So
you
have
a
cluster
which
is
running
or
spanning
across
multiple
data
centers,
and
in
that
case,
what
you
try
to
do
is
it's.
It's
always
recommended
that
you
run
not
odd.
Sorry,
not
even
with
odd
number
of
sides,
so
not
two
sides,
but
at
least
have
three
sides,
because
each
cd
needs
like
quorum
and
it's
column
of
three
and
it's
better
to
have
three
sides.
So
if
you
lose
one
side
still
the
major
of
the
quorum
will
be
available
and
your
cluster
will
be
operational.
B
So
in
this
case,
what
you
try
to
do
is
that
you
try
to
build
up
a
cluster
where
you
have
multiple
sites.
So,
for
example,
we
let's
say
you're
doing
it
on-prem,
and
then
you
will
have
multiple
on-prem
sites
where
you're
trying
to
replicate
the
data,
and
in
our
experience
like
this
one,
this
this
strategy
is
it
works
really
well
as
well,
and
what
you
have
to
keep
in
mind
is
that
you
need
to
have
enough
capacity
on
the
other
sides
if
one
of
the
sites
goes
down,
so
that
is
then.
B
This
goes
back
to
your
capacity
planning,
but
it
is,
of
course
better
than
having
two
separate
clusters.
It's
it's
it's
better
than
that
that
you're
having
then
two
separate
clusters,
one
is
active
and
one
is
is
not
active
so
once
and
in
in
that.
So
in
this
case
the
whole
cluster
is
stretched
over
multiple
sides,
like
most,
let's
say,
stretched
over
two
or
three
sides
and
in
that
case
you're
also
relying
on
the
underlying
infrastructure.
B
So
in
that,
if
you,
if
you
lose
one
site,
then
you
would
expect
your
vmware
technology
and
your
that's,
not
vm
technology,
not
in
vmware,
sorry,
vm
technology,
whatever
you're
using
and
your
data
storage
technology,
whatever
you're,
using
to
ensure
that
the
data
is
being
replicated
to
all
the
sites
and
the
where
two
machines
can
also
move
when
one
side
goes
down.
So
we
have
done
experiment
with
this
setup
with
customers
to
to
where
they
had
to
test
out
their
vr
strategies.
B
Where
you
take
out
the
one
one
of
the
one
of
the
sides
and
the
underlying
technology
that
you're
using
that
moves
stuff
quite
seamlessly.
So
you
don't
you
don't
see
any
difference
like
okay,
I
lost
a
site,
but
since
the
other
side
had
enough
of
the
capacity
things
were
kept
kept
running
and
but
but
you
know,
I
think
one
very
important
thing.
B
Unless
you
have
underlying
technology
that
will
bring
back
those
the
the
the
two
masters
in
the
broken
site
to
your
other
side.
So
because,
as
long
as
your
ip
and
host
name,
they
stay,
the
same
kubernetes
will
not
notice
that
this
master
has
has
moved
from
one
place
to
another
place
because
for
it
it's
seamless.
B
B
So
we
have.
We
have
used
this
strategy
quite
a
lot
and
it
works
very
well
and
now
we
have
been
recently
helping
out
or
doing
some
stuff
where
we
have
two
on-prem
sites
and
a
third
set.
The
third
side
is
cloud,
so
that's
quite
common
scenario
as
well,
so
that
you
add
a
third
site
here
where
the
third
site
is
your
cloud.
So
you
have
two
public
sorry,
two
on-prem
sites
and
the
third
is
a
site
on
a
cloud,
and
then
you
can
scale
out
on
the
cloud
that
you
need
on
demand.
B
B
And
then
then
it
comes
back
to
the
to
the
last
strategy.
That's
called
rebuild
strategy,
and
this
is
a
really
nice
strategy,
I'm
pretty
sure
if
you
are
familiar
with
with
kubernetes
or
or
stuff
like
that,
you
might
have
heard
about
gitops
kind
of
getting
very
famous
these
days,
and
the
idea
is
that
your
git
is
a
single
sort
of
shoot
for
your
declarative
infrastructure.
B
Every
entire
system
is
described,
declaratively
and
self.
Everything
is
like
in
a
git.
You
can
see
stuff
in
the
git
like
like
no
get
get
divs
and
you
can
roll
back
and
you
can
do
all
that
stuff,
which
is
pretty
nice.
So
what
the
idea
with
this
is
that,
rather
than
doing,
rather
than
what
you
say,
repairing,
why
not
rebuild
the
cluster
so
that
that
works
pretty
well
as
well.
So
if
you
have
a
complete
automation,
rebuilding
is
something
faster
than
actually
repairing
a
cluster.
B
But
again
you
will
have
to
keep
in
mind
your
rpo,
your
rto
and
your
persistent
volumes.
So
whenever
talking
about
these
kind
of
stuff,
like
git
ops,
the
whole
idea
is
that
this
cluster
state
is
in
a
git
wrapper.
So
you
don't
need
to
take
an
hcd
backup
anymore,
because
whatever
you
deploy
to
the
cluster
is
coming
from
a
git
wrapper.
So
it
means
if
I
lose
a
cluster
completely.
B
I
don't
have
to
worry
about,
because
I
can
bring
up
a
new
cluster
and
I
can
apply
the
same.
Git
git
wrapper
there
and
it
will
have
exactly
the
same
number
of
things
running
all
the
time.
Well,
I
mean
the
next
time
but
again
for
stateless
applications.
It's
gonna
work
perfectly
awesomely
and
you
can
do
it
very
quickly
like
rp
and
audio.
B
Then
you
have
to
combine
this
with
your
your
storage,
backup
recovery
strategy
as
well,
but
of
course
you
can
think
about
of
having
your
hcd
stuff
or
in
in
a
git,
ops
or
apple.
But
then
you
need
to
have
a
mentality.
What
we
call
everything
is
stored,
so
your
infrastructure
has
to
be
as
code,
which
includes
cloud
configurations,
vm
configurations,
even
the
kubernetes
cluster
setup.
B
Sorry
and
your
deployments,
your
tools,
applications
configurations,
so
they
all
everything
has
to
be
in
code.
So
when
you
talk
about
cloud
infrastructure
s
code,
so
basically
you
have
a
pipeline
and
that
pipeline
basically
first
brings
up
your
cloud
infrastructure
like
compute,
storage
and
security.
Then
you
have
next.
That
brings
up
your
equivalence
cluster
on
the
cloud
another
step
in
the
pipeline.
You
have
a
third
step
which
deploys
your
what
we
call
sporting
tools,
like
maybe
premiere
tears
or
whatever
you're
using
efk.
B
B
So
if
I
try
to
summarize
what
we
looked
into,
I
think
one
thing
that
is
very
important
is
to
plan
ahead
so
review
your
disaster
recovery
requirements
and
what
is
the
tolerance
level
of
data
and
service
loss
that
you
can
bear?
What
should
be
your
rpo
and
rto?
So
just
make
sure
that
you
you
plan
ahead
of
it
and
then
once
you
have
you
those
matrices,
then
you
look
into
your
workload.
Okay,
what
workload
am
I
running?
Am
I
just
running
stateless
workloads,
okay,
yeah,
I'm
running
only
stateless
workload.
Then
I
have
a
I
have.
B
I
can
follow
this
strategy
now.
I'm
running
a
stateful
workloads
with
also
like
I'm
running,
not
only
stateless
but
also
straightforward
workloads.
Then
I
have
to
think
about.
Where
am
I
running
my
workload?
Is
it
on-prem?
What
underlying
technologies
am
I
using?
How
do
they
support
me
in
the
in
backup
of
my
of
my
data,
and
then
you
plan
that
stuff
accordingly,
and
then
you
should,
you
know,
create
backup
and
recovery
procedures
to
best
meet
your
specific
requirements.
B
So
so
look
at
your
requirements
and
then
you
can
look
from
these
different
strategies
that
I
mentioned
that
which
one
will
fit
best
for
your
for
your
needs,
and
this
is
very
important.
Don't
just
plan
and
document
ensure
that
you
also
practice
this
in
real
and
and
and
this
practice
is,
should
not
be
like
one
time.
This
practice
has
to
be
continuous,
like
you
should
think
it
in
a
way
where,
for
example,
what
we
do
with
our
a
lot
of
I
mean
in
our
own
environments,
where
we
run
manage
openshift
or
kubernetes
for
customers.
B
We
actually
do
a
lot
of
times.
We
like,
for
example,
if
they
are
like
dev,
test
environments
or
dev
test
clusters,
we
actually
destroy
them
every
night:
okay,
let's
just
delete
them
every
night,
and
then
we
have
procedures
in
place
that
next
day
in
the
morning,
you
have
a
new
cluster.
Everything
is
brought
back
to
the
same
state,
and
so
you
have
a
continuous
testing.
B
It's
been
happening,
but
not
like
once
in
a
blue
moon
that
you
test,
because
then
your
procedures
will
get
very
quickly
outdated
because
a
new
technology
was
introduced
and
then
no
one
looked
into
that
and
then
now,
when
you
try,
things
are
failing.
So
it
has
to
be
a
continuous
part
of
your
ongoing
day-to-day
job.
That's
a
workflow,
that's
very
important
and
you
can
do
this
kind
of
stuff
in
your
dev
test
environment.
B
Then,
of
course,
you
can
look
into
what
we
call
chaos,
engineering
tools
like
class
monkey
and
those
kind
of
stuff.
So
what
what
we
do
a
lot
is
is
basically,
if
you're
running
a
public
clouds,
we
use
what
we
call
spot
instances.
So
spot
instances
is
a
built-in
class
monkey
for
us,
because
you
lose
a
sport
instance
and
things
have
to
recover
back.
So
it's
like
you're
continuously.
Having
I
would
say
a
built-in
test
is
testing
is
happening.
Of
course,
we
don't
run
masters
on
the
spot
instances.
B
Only
the
worker
machine
nodes,
we
try
to
run
on
spot
instances,
and
then
we
can
test
it
like
built
in
in
that
you,
you
can't
necessarily
test
it
and
then
it's
it's
very
important
that
you
should
consider
failures
on
every
layer.
Remember
I
mentioned
that
I
can.
I
like
kubernetes
disaster
recovery
strategy,
might
rely
on
your
underlying
infrastructure,
your
vm
technology
and
your
data
storage
technology.
B
It
might
fail
to
do
the
backup
it
might
fail
to
restore
like
help
you
in
stress
or
similarly
things
can
happen
on
your
vm
layer,
like
your
virtual
machines,
can
fail
and
might
not
be
able
to
recover.
So
just
just
look
into
that
very
thoroughly
and
test
as
well,
so
like
masters,
workers
and
of
course
stories.
B
So
now
I'm
done
with
the
presentation,
and
I
would
say
thank
you
very
much
everyone
for
listening
me
today
and
we
are
very
passionate
about
kubernetes.
So
I
hope
you
have
questions.
I
see
there
are
a
few
questions
out
there
already.
Let's
see
if
I
can
answer
them
or
not,
but
let's
I
will
try
my
best
so.
A
A
Yeah,
thank
you
very
much
rasheed
for
that
wonderful
presentation.
We
do
have
a
few
questions
in
here
already,
but
we
will
have
about
10
minutes
for
q
a
so.
If
anyone
else
has
any
questions,
please
feel
free
to
drop
them
in
first
question:
if
I'm
running
my
k8
cluster
in
the
cloud,
let's
say
aws
and
I
take
a
snapshot
of
the
of
the
etcd
volumes
mounted
to
the
master
vms.
A
B
Yeah
right,
so
I
think
in
in
that
case
it's
like
it's,
it's
not
ecd
volumes.
I
would.
I
would
prefer,
in
that
case
snapshots
of
the
ecd,
the
built-in
ecd
snapshots
and
not
the
snapshots
of
the
of
the
volume.
But
of
course,
if
you
have
snapchat
the
volume-
and
if
you
have
these
things
like
ips
and
host
names,
what
you
say
being
code
decoded
into
the
setup,
then
it
doesn't
work.
B
You
know
because,
as
I
mentioned,
if
you
want
to
recover
and
you
need
cd
to
recover
from
a
state
it
needs
and
where
you
have
lost
few
members
of
the
quorum
and
not
everything
else,
then
they
need
to
have
same
host
name
and
ip.
If
they
don't
have
same
host
in
ip,
you
can
still
recover.
But
then
you
have
to
do
a
lot
of
manual
work
in
between
to
remove
nodes
from
your
hd
cluster
and
add
back.
B
Yeah,
it's
another,
a
billion
dollar
question,
and
so
I
mean
we
again
here.
You
have
multiple
options
like
if
you're,
using
it
ops
approach
how
you
handle
secrets.
So
we
are
using
different
approaches
here.
One
of
the
solutions
that
we
use
quite
a
lot
is
seal
secrets.
It's
a
it's
quite
okay,
it
works.
But
again,
in
that
case
you
have
to
like
seal
your
secret
for
each
cluster
depending
on
that's
how
it
works,
but
we
have
been
also
using
the
hashicrop
vault
open
source
one
and
that
also
works
pretty
nicely.
B
And
then
we
use
things
like
gopass
to
put
up
the
shared
secrets.
If
the
developers
need
to
know
those
secrets,
but
we
never
recommend
to
put
up
the
plain
secrets,
vanilla
secrets
out
in
into
your
github
strapper:
that's
never
never
recommended,
and
you
should
never
do
that.
So
both
if
you
want
to
have
a
lean
start,
then
field
secret
is
a
good
option.
It
works
pretty
nicely
fits
very
nicely
into
your
into
your,
I
would
say
github's
workflow,
but
again
that
really
doesn't
solve
the
problem.
That
is
my
secret.
Secure.
B
Now
no
see
secrets
is,
doesn't
secure
your
secret
because
still
I
can
get
into
my
into
my
cluster
and
I
can
read
the
secret.
So
if
you
wanna,
if
you're
really
conscious
about
your
secrets,
then
you
have
to
think
about
a
solution
that
gives
you
secrets
on
the
fly
and
puts
it
in
in
a
location
that
you
can't
get
in
and
read
from
as
a
human
user.
B
C
B
So
I
think
in
in,
if
I'm
running
a
public
cloud
I
would
in
a
region
I
will
go
multiple
availability
zones.
I
will
run
my
cluster
in
multiple
availability
zones.
So
that's
one
thing
that
I
I
would
do
I
mean
not.
I
don't
you
don't
need
to
go
to
multi-region,
because
multiple
availability
zones
are
are
perfectly
fine
in
that
case,
and
if
you
are
actually
running,
as
I
mentioned,
stateful
workloads
you,
you
can
use
look
into
the
open
source
tools
like,
for
example,
velero.
B
B
In
what
forms-
and
I
think,
as
I
answered
before-
look
into
sealed
secrets
now-
there's
a
perfect
for
github's
approach.
Seal
seekers
works
perfectly
fine
and
you
will
like
the
you
will
like
the
workflow,
so
we
put
like
in
in
a
in
a
github
strap
as
a
seal
secret.
But,
as
I
mentioned,
if
you
want
to
go
really
you're
conscious
about
your
secrets,
then
you
need
to
look
into
other
options.
B
Where
you
don't
put
your
secrets,
you
know
you
don't
put
your
secrets
in
a
git
wrapper,
but
rather
in
some
other
tool
that
takes
care
of
your
secrets
handling.
But
if
you
have,
let's
say
secrets
which
are
pretty
like
commonly
shared
by
developers,
then
I
will
suggest
you
to
look
in
something
open.
Source
called
go
pass.
So
that's
very
nice
tool.
So
that's
how
we
share.
B
Within
a
team
we
use
gopass
to
share
the
common
passwords,
because
in
gopass
you
need
to
have
multi,
multiple
signers
to
be
able
to
get
access
and
read
a
secret,
so
yeah.
So
that's
a
way,
but
that's
not
like
a
part
of
github
strategy.
It's
more
about.
We
use
this
third-party
system
called
abc,
and
I
would
like
we
as
a
developer
team
would
like
to
know
what
is
the
password?
B
A
B
I
think
if
you
have
stateless
services
and
you're
going
with
eks
your
your
best
strategy
is
go
with
github
from
day
one,
and
then
you
will
have
your
your
your
your
ds.
They
are
built
in
actually
and
how
I
would
we
would
do
it.
I
would
let's
say
I'm
going
with
eks
and
I
have
two
clusters.
One
is
dev
test.
One
is
production.
B
I
would
just
have
two
separate.
I
will
have
a
git
ops,
repo
and
I
will
have
everything
is
stored.
I
will
delete
the
well.
I
will,
but
we
do
that
as
well
with
our
self
as
well.
We
delete
the
dev
test
cluster
every
night
and
or
and
over
the
weekends.
That
also
saves
us
cost,
and
then
we
have
tools,
the
pipelines
that
bring
the
cluster
back
and
then
the
same
gitoffs
or
repos
is
applied
back.
So
when
you're
looking
into
gitobs,
you
will
look
into
the
cncf.
B
I
think
both
projects,
flux
and
argo
cd
we
have
used
both
of
them
and
flux
is
more
like
very
lean
start
like
because
it
does
nothing
more
than
cube
cut
apply.
But
if
you
want
to
go
a
bit
fancier
with
nice
ui,
you
can
look
into
orgo
cd
and
you
can
deploy
both
of
them
as
code
as
well
so,
which
makes
it
very
easy.
So
I
bring
up
a
new
cluster,
I
will
deploy
flux,
tell
him
which
record
to
watch
and
bank.
B
I
have
a
complete
my
dr
strategy
built
in
you
know
so
for
stateless
applications.
In
our
experience,
if
you
have
only
status
services,
git
ops
is
the
best
strategy
for
your
dr.
Don't
look
into
anything
else.
Much
faster
bring,
I
mean
rebuild,
not
repair,
just
kill
the
cluster.
I
mean
we
have
done
that.
We
just
kill
the
clusters
and
bring
new
thing
back
and
sometimes
it's
much
faster
than
repairing
a
cluster.
Because
for
repairing
a
cluster
you
will
go
and
read
documentation.
B
I
mean
I
mean
if
you're
starting
flux
is
much
better,
very
lean,
simpler.
It's
not
that
I
like
it
more,
it's
more
like
it's.
It
does
not
introduce
you
any
new
concepts,
because,
as
I
mentioned,
flux
is
only
doing
cubes,
it
will
apply
so
it's
much
faster.
But
when
you
go
into
argo
cd,
then
you
have
a
bit
of
a
more
learning
curve
because
you
have
to
first
learn
because
it
introduces
some
new
concepts
like
an
application
and
those
kind
of
repository.
So
you
have
to
learn
with
new
more
things.
B
But
if
you
want
to
do
github,
you
are
just
starting
with
then
flux
will
take
you
literally
very
far
away
and
a
lot
of
how
we
do.
We
do
flux
for
the
we
do
flux
for
the
infrastructure
and
then
we
use
parco
cd
for
for
for
applications.
So
we
do
a
lot
of
that
combination
as
well,
where
we
are
using
both
flux
for
deploying
infrastructure
when
I
say
deploying
premiere,
tiers
and
those
tools
and
argo
cd
when
deploying
the
developer
applications.
B
B
Yeah
I
mean
it's
it's
it's
again,
a
good
question
and
the
load
balancers
are
usually
considered
like
outside
of
your
because
you
have
your
dns
entries
pointing
to
your
load
balancers
and
then
from
load
balancers.
The
traffic
gets
into
your
into
your
clusters.
So
when
I
talked
about
the
strategies
we
the
load
balancer
is
is
running
outside
of
the
picture.
It's
not
part
of
the
pictures
it's
outside
and
that
does
not
get
affected
as
long
as
you
have
same
dns
entries
and
it
can,
it
can
point
to
the
to
the
same.
B
What
should
say
the
load
balancer
ip,
but
of
course,
if
you're
bringing
up
a
new
cluster
completely,
then
you
will
have
to
go
and
change
your
your
load,
balancer
setting
to
point
to
the
new
ip
of
your
load,
balancer,
and
so
again
I
think
it
again
it
will
depend
you
know
which
strategy
you
are
going
for
you're
working
with.
Then
it
will
affect
your,
I
would
say,
rp
and
rto.
In
that
sense,.