►
From YouTube: Clusters as Cattle: How to Seamlessly Migrate Apps Across Kubernetes Clusters - Andy Goldstein
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Clusters as Cattle: How to Seamlessly Migrate Apps across Kubernetes Clusters - Andy Goldstein, Heptio (Intermediate Skill Level)
Before the arrival of Cloud Native, IT departments frequently treated each component--a service/application, a virtual machine, or a bare metal server--as a special, fragile entity that required the utmost of care. Kubernetes, and more broadly Cloud Native, presents us with better ways to handle our infrastructure. For example, when we need to upgrade to a newer Kubernetes version, we can use automation and tooling to create a new cluster and migrate existing workloads over to it. In this talk, Andy will describe different strategies for moving workloads between clusters. He'll show you how to use tools such as Ansible and Kubeadm to quickly install a new cluster, along with Heptio Ark to back up one cluster and restore into a new one. Andy will also demonstrate how you can perform zero-downtime migrations using Envoy for cluster ingress, traffic shifting, and some DNS “magic.”
About Andy
Andy Goldstein is an engineer at Heptio where he works on tooling to make operating Kubernetes clusters easier, such as Ark, a disaster recovery tool for backing up and restoring Kubernetes workloads and persistent data. He is also a contributor to Kubernetes. Prior to his current role, Andy worked on Kubernetes and OpenShift at Red Hat. Andy lives in Rockville, MD, with his wife, two children, and two noisy cats.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
everyone
thank
you
for
coming.
This
is
clusters
as
cattle
how
to
seamlessly
migrate
your
apps
across
your
kubernetes
clusters.
My
name
is
Andy
Goldstein
I
work
at
hep
do
I've
been
a
kubernetes
contributor
for
the
past
several
years,
I'm
the
tech
lead
for
hep
do
arc,
which
is
our
backup
and
recovery
tool
for
your
kubernetes
resources
and
persistent
data.
So
what
are
we
gonna
talk
about
today?
A
We're
going
to
talk
a
little
bit
about
a
history
of
developing
and
deploying
apps
over
time,
we'll
talk
about
kubernetes
and
how
the
pets
versus
cattle
metaphor
fits
in
there
I'll
go
over
some
app
and
workload,
migrations
in
kubernetes
and
I'll
show
you
a
little
demo
that
I
hope
you'll
enjoy
and,
at
the
end,
we'll
have
some
time
for
questions.
So
if
you
were
in
the
1940s
or
50s
and
you
were
deploying
an
app,
this
is
what
it
would
look
like.
This
is
a
picture
of
ENIAC
one
of
the
first
electronic
computers.
A
It
took
up
an
entire
room
or
more,
and
if
you
needed
to
develop
an
app
or
and
deploy
it,
you
physically
had
to
rewire
to
make
your
changes
moving
forward.
A
little
bit
we
have
some
punch
cards
so
better
than
physically
rewiring,
but
not
exactly
readable
to
a
normal
person,
and
it
takes
a
lot
of
time
to
both
program
and
when
you're
ready
to
deploy
your
app,
you
have
to
run
it
through
a
punch
card
reader,
like
you,
can
see
here
with
this
IBM
704
electronic
calculator
fast
forward
past.
A
All
of
that
past
mainframes
and
we
get
to
things
like
UNIX
and
Linux
and
Windows,
and
now
developing
apps
is
easier,
deploying
apps
a
little
bit
easier,
but
getting
access
to
those
computing
resources
still
isn't
super
easy.
What
you
have
to
do,
at
least
back
then,
is
file.
Tickets
I
need
a
new
server.
I
need
a
new
virtual
machine.
I
need
something
deployed
and
with
tickets
comes
delays,
you
have
to
get
things
approved.
Maybe
multiple
people
have
to
approve
it
so
we're
better
than
having
to
rewire
physically,
but
we're
still
not
there
yet.
A
This
is
your
pet
because
you
never
know
how
long
it's
going
to
take
to
get
this
thing
back
working,
so
you've
got
to
get
it
up
as
quickly
as
possible.
You
want
to
make
sure
that
you
have
the
latest
operating
system
updates
and
you
want
to
make
sure
that
if
something
does
go
wrong,
you
can
recover
from
your
backup
as
quickly
as
possible.
Okay,
obligatory
container
picture
so
beyond
cloud
computing
with
virtual
machines,
we
have
our
containerization
technologies,
which
are
great.
We
don't
have
to
think
about
the
operating
system
nearly
as
much.
A
We
still
do
but
at
least,
if
you're
developing
an
image
for
your
application,
you
can
focus
on
just
what
you
want
to
fit
in
there.
So
if
you're
working
on
a
web
application,
you
put
your
web
server
and
you're
content
most
likely,
you
package
it
up
as
an
image
and
then
you
can
run
it,
which
is
great.
You
don't
have
to
worry
about
installing
an
operating
system
to
make
this
happen,
but
when
it
comes
to
orchestrating
where
these
containers
run,
it's
still
difficult,
maybe
you
have
systemd
or
upstart
or
some
other
init
system.
A
You
can
tell
it
run
this
container
on
this
node
on
this
server,
but
from
a
management
and
orchestration
perspective,
you're
still
not
there.
So,
fortunately
we
have
kubernetes.
That's
why
we're
all
here
so
the
great
thing
about
kubernetes
is.
You
can
now
really
forget
for
the
most
part
about
that
infrastructure
and
where
things
are
running,
we
tell
kubernetes
or
more
ask
kubernetes,
deploy
something
I,
don't
really
know
where
you're
going
to
put
it
because
you
get
to
schedule
it.
A
A
So
ansible
is
a
general
purpose
tool
for
running
collections
of
tasks
that
have
a
similar
purpose.
You
group
those
into
roles
and
then
you
can
have
multiple
roles
in
an
individual,
playbook
and
I'll
have
an
example
here.
So
you
can
see
how
that
looks.
So
the
first
thing
I
want
to
show
is:
this
is
a
role
that
I've
called
kubernetes
and
it
will,
if
I
run
it,
create
a
one
node
all
in
one
cluster.
A
Obviously,
that's
a
bit
limiting,
but
for
this
purpose
of
this
talk,
I
think
you
can
get
the
point
and
then
you
can
go
and
figure
out.
There's
great
examples
out
there
for
doing
multiple
node
clusters
and
for
adding
additional
nodes
to
existing
clusters.
And
what
you
see
here
is
a
default
variables
file
for
things
that
maybe
I
want
to
just
have
by
default,
my
kubernetes
version
and
if
I
don't
override
it.
This
is
what
I'm
going
to
get
once
I
have
that
set
I
have
some
tasks.
A
These
are
the
real
meat
of
what
happens
when
I
run
this
role.
So
the
first
thing
I
need
to
do
is
get
my
packages
installed.
I
need
cube,
control,
the
cubelet
cube
ATM
and
kubernetes
CNI
I'm
using
fedora,
so
I've
got
DNF
up
there
and
when
I
run
this
task,
it
will
make
sure
that
these
four
packages
are
present
on
the
machine,
and
the
nice
thing
about
ansible
is
that
it
has.
A
A
So
you
can
see
up
here,
there's
a
templating
language
that
allows
ansible
to
take
either
my
default
value,
which
was
1.9
point
6
or
a
version
of
my
choosing
to
override
and
replace
that,
so
that
I
can
change
the
kubernetes
version
that
I'm
installing,
without
necessarily
having
to
rewrite
my
role.
You'll
also
see
that
I
have
a
pod
subnet
for
calico
and
I've
decided
that
I
want
to
use
core
DNS
as
well
once
I
have
that
I
can
use
the
template
command
to
run
it
through
the
template
engine.
A
So
you
can
go
check
this
out
and
play
around
with
it
on
your
own.
After
this
talk
and
finally,
to
put
all
of
this
together,
I
have
a
playbook,
and
in
this
case
my
playbook
is
called
cluster
and
I
want
to
run
two
roles
with
this
playbook.
The
first
one
I
just
took
you
through
some
examples
of
the
second
one
installs,
the
contour
ingress
controller,
which
will
be
part
of
a
demo
that
I'll
be
showing
in
a
minute.
A
Ok,
so
you've
got
your
cluster
installed
and
you've
automated
your
installation.
So
let's
talk
about
applications.
Hopefully
you
have
it
your
applique
configurations,
whether
it's
yeah,
Mille
or
case
on
it
or
something
else.
Hopefully
you
have
that
in
source
control,
that's
very
important
and
something
that
we
stress
highly
that
you
do
and
but
once
you
have
your
applications,
hopefully
automated
you
may
need
to
migrate
them
at
some
point.
So
if
you've
got
an
application,
you
can't
think
that
you're
going
to
pin
it
to
a
specific
cluster
or
a
specific
node
in
a
specific
cluster
forever.
A
At
some
point,
the
node
is
going
to
be
decommissioned
or
the
pods
just
going
to
get
killed
or
the
cluster
needs
to
be
decommissioned.
Maybe
so
we
need
to
think
that
migrations
are
necessities
and
we
have
to
accept
that
and
account
for
that.
So
when
we're
doing
migrations,
stateless,
obviously
is
easier
than
staple.
You've
got
an
app
that
doesn't
have
any
state.
It's
pretty
pretty
easy
to
move
it
around
with
stateful
apps
I'd
say
that
probably
depends
on
the
app
itself,
and
so
I
can't
give
you
a
generic.
A
This
will
all
work
for
every
single
stateful
app,
but
you
can
do
things
such
as
maybe
put
it
in
read-only
mode.
If
you
can
accept
that
and
your
users
can
accept
that,
maybe
you
have
to
take
it
offline
for
some
downtime
or
maybe
you
have
some
apps
that
can
support
spanning
multiple
clusters,
with
replication,
for
example,
so
that
you
can
start
to
spin
up
new
members
in
a
new
cluster
and
bring
your
old
members
down
over
time.
A
So
I
do
want
to
mention
I
said
earlier
that
I'm
the
tech
lead
for
arc
and
it's
good
for
disaster
recovery.
But
it's
also
really
good
for
migrations.
So
we
do
on-demand
and
scheduled
backups
with
arc
and
we
backup,
as
many
or
as
few
kubernetes
resources
as
you
specify,
as
well
as
persistent
volume
data.
So
you
can
say
I'd
like
to
back
up
one
namespace
or
a
bunch.
A
You
can
say
I'd
like
only
a
certain
subset
of
resources,
so
maybe
just
pods
and
services,
and
you
can
also
use
a
label
selector
as
part
of
that
selection
criteria
when
you're
creating
backups
we're
also
extensible.
So
if
we
don't
have
the
exact
behavior
that
you
need,
you
can
write
a
plug-in
for
it
and,
as
I
mentioned
it's
good
for
migrations,
so
you
can
take
a
backup
of
something
in
one
cluster,
including
persistent
data
and
then
migrated
over
to
another
cluster.
A
So,
let's
look
at
a
scenario
here:
let's
say
that
I
have
an
app
and
it's
running
in
a
kubernetes
1.9
cluster
and
I
want
to
migrate
to
one
point
10,
but
I
don't
want
to
upgrade
my
1.9
cluster
I'm
afraid
that
maybe
something
goes
wrong
and
I
end
up
with
a
non-functional
application
or
a
non-functional
cluster,
so
to
avoid
downtime.
What
I
want
to
do
is
keep
my
old
cluster
spin
up
a
brand
new
cluster
using
the
new
version
and
then
slowly
migrate
users
from
the
old
one
to
the
new
one,
but
do
it
transparently.
A
So
they
have
no
idea
that
this
is
happening
now
before
I
get
into
that
and
the
demo
I
do
want
to
talk
a
little
bit
about
networking
inside
of
kubernetes,
because
it's
relevant
to
how
we
can
solve
this
migration
problem.
So
if
you're
writing
a
client
and
it's
running
inside
the
cluster-
and
you
want
to
talk
to
another
pod-
you
can
talk
to
the
pod
by
its
IP
address.
So
you
could
hard
code
that
a
better
thing
to
do
is
to
use
the
DNS
name
of
the
service.
A
So
you
put
a
service
in
front
of
the
pod:
it
gets
a
DNS
name.
It
also
has
an
IP
address.
You
could
use
that
if
you
wanted
to
and
if
you're
not
familiar
with
services
and
kubernetes,
they
have
endpoints
that
are
associated
with
them
and
there's
an
endpoint
for
the
pods
that
are
working
for
that
service.
So
this
works
great.
A
But
what
happens
if
I
I
lose
my
pod
and
my
deployment
replaces
it
with
another
one:
I
have
a
new
IP
address,
so
if
I
hard-coded
that
it's
not
going
to
work
anymore,
fortunately
DNS
still
works
and
that's
what
I
would
strongly
recommend
if
you
have
an
in
cluster
client.
But
what,
if
I,
have
an
outside
client
like
someone
from
the
internet,
I'm
not
going
to
be
able
to
hit
this
internal
cluster
only
IP
and
I'm
not
going
to
be
able
to
hit
the
DNS
for
the
service
either
because
that's
local
to
the
cluster.
A
A
I
changed
the
configuration
a
little
bit,
but
I
can
remove
and
add
back-end
clusters
at
will
and
as
long
as
there's
at
least
one
running,
my
traffic
will
continue
to
flow
and
my
app
will
continue
to
function.
So
how
do
we
do
this?
On
the
right
here?
I've
got
a
representation
of
just
a
back-end
cluster
node
I
have
contour
and
Envoy
running
so
that
if
I
were
to
hit
that
hit
on
void
directly
I'd
be
able
to
route
it
into
that
pod.
But
this
is
not
exposed
to
the
Internet.
A
A
So
what
I
have
is
some
custom
control
logic
that
will
look
for
special
secrets
that
are
labeled
in
a
certain
way
and
inside
each
secret.
I
have
two
things:
I
have
an
IP
address
that
represents
the
IP
of
this
node,
that's
running
contour
and
envoy,
and
a
back-end
cluster
and
I
have
a
cube
config
file
that
I
use
to
talk
to
that
back-end
cluster,
because
what
I
want
to
do
is
I
want
to
find
all
of
the
ingress
objects
that
have
a
route
equals
to
label,
and
this
is
all
just
an
example.
A
You
know
we
can
do
this
different
ways
but,
for
example,
purposes
I'm
looking
for
route
equals
true
and
when
I
find
that
when
my
controller
finds
it,
it
adds
a
service
and
an
endpoints
object
that
are
specific
to
that
ingress
in
the
backend
cluster
and
the
endpoints
object
here.
Has
one
IP
address
in
it
that
points
at
that
envoy
running
in
that
back-end
node
and
it
updates
the
ingress
so
that
I
can
actually
route
traffic
appropriately.
A
So
here's
what
that
looks
like
if
I
come
to
QWERTY
demo,
it's
going
to
hit
this
Envoy
up
here
and
envoy
is
going
to
select
a
back-end
based
on
the
endpoints
that
I
have
here
which
point
to
the
node
in
the
backend,
and
then
we
flow
all
the
way
down
to
that
pod
so
demo
time.
Hopefully
the
demo
gods
are
with
me
all
right.
A
So
let's
go
ahead
and
we're
going
to
deploy
quar
DV
1
to
cluster
1,
so
I'm
going
to
create
a
namespace
called
demo
and
I'm
going
to
use
the
cube
control
run
command
to
run
this
QWERTY
image,
I'm
also
going
to
expose
it,
which
will
create
a
service
for
me
on
port
8080,
and
so
here
you
can
see
that
output
and
then
I
have
an
ingress
file
that
just
has
cardi
demo
pointing
to
this
particular
service.
So
that's
all
created.
A
We
can
go,
take
a
look
in
the
browser
and
if
we
go
over
here,
this
is
QWERTY
demo
I'll
refresh
it
it's
not
working
because
I
haven't
hooked
everything
up
yet
so
the
next
thing
we
want
to
do
is
we
want
to
tell
my
routing
cluster
about
this
website
that
I
want
to
have
hosted
so
I
have
a
little
helper
command
after
I,
create
this
namespace
here
called
R
and
I'm
going
to
add,
AV
host
called
Cordy
and
I
have
some
defaulting
in
here.
So
this
is
basically
going
to
say:
let's,
let's
do
QWERTY
demo.
A
So
what
does
this
actually
do?
This
creates
an
ingress
and
I've
abbreviated
a
little
bit,
but
you
can
see
that
there's
a
namespace
demo.
The
name
is
Cordy
and
I
have
an
annotation
here
called
weighted
cluster,
and
this
is
just
something
that
I
did
for
the
demo
to
tell
envoy
that
we're
going
to
be
doing
some
traffic
shifting
and
in
particular
I'll
point
out,
you
can
see
I
have
this
temporary
placeholder
up
here
for
my
back-end
service.
This
is
because
I
haven't
told
the
router
about
any
of
my
back-end
clusters.
A
So
let's
go
ahead
and
do
that
so
if
I
add
my
back-end
cluster
1,
we
can
see
what
that
did
to
the
ingress
itself.
So,
instead
of
temporary
placeholder,
we
now
have
this
quality
cluster
1
service,
which
my
routing
controller
created
automatically
for
me
when
I
added
that
back-end
to
it
and
if
I
go
into
the
website-
and
I
refresh
it-
I
now
have
the
app
up
and
running
and
I
can
refresh
a
couple
times
and
you'll
see
that
down
at
the
bottom.
A
This
request
ID
changes
here
and
there,
so
we
just
have
one
back-end
cluster
and
the
application
deployed
there
all
right.
So,
let's
hit
hit
it
with
some
load,
so
I
have
a
little
benchmarking
script.
That's
just
going
to
run
a
five
second
load
test
against
QWERTY
demo,
and
this
will
show
us
how
many
requests
came
in
and
which
back-end
they
hit.
So
when
this
pops
up,
we
had
about
30
400
requests
and
they
all
went
to
this
one
particular
IP
address,
which
is
my
cluster
one,
all
right.
A
So,
let's
create
a
backup
with
ARC,
so
I'm
gonna
go
over
to
cluster
one
I
have
arc
pre-installed
and
I'm
gonna
use
the
arc,
backup
create
command
to
tell
it.
I
want
to
create
a
demo
or
sorry
I
create
a
backup
called
quar,
D
and
I
want
to
include
one
namespace,
which
is
the
demo
namespace.
So
this
submits
a
request
to
the
server
the
arc
server
which
will
find
it
and
process
it.
We
can
go.
A
Take
a
look
I'm,
storing
this
in
Mineo
and
here
you'll
see,
I
have
a
bucket
called
arc
and
I
have
a
backup
called
Cordy,
and
this
is
what
you
typically
see
inside
for
an
arc
backup.
So
that's
all
good
and
we
can
go
ahead
and
lo
and
behold
we
have
a
brand
new
kubernetes
one
10.14
us.
If
I
had
more
time,
I
would
I
would
spin
this
up
with
vagrant
and
run
ansible
on
it.
But
I
don't
want
to
make
you
sit
here
and
wait
for
that
to
happen.
A
So
I
have
this
cluster
pre
created,
so
we're
gonna
go
ahead
and
install
arc
and
this
installs
arc
in
restore
only
mode.
So
it's
going
to
connect
to
the
same
Mineo
bucket
that
my
original
cluster
was
using,
but
it
won't
try
and
delete
any
backups
for
garbage
collection
or
anything
like
that.
So
it's
it's
purely
restore
or
read-only
mode.
A
Alright,
one
of
the
things
that
arc
does,
if
I
have,
if
I
have
backups
in
a
bucket
and
I
point
a
new
cluster
at
that
same
bucket,
all
those
backups,
the
metadata
about
them
gets
synced
into
the
new
cluster.
So
here
you
can
see
I'm
in
cluster
two,
and
it
says:
I
know
that
this
backup
called
cardi
exists,
which
is
great.
This
is
the
migration
part.
So
let's
say
that
I
want
to
go
ahead
and
do
a
migration
first,
we'll
check.
Let's
make
sure
that
I'm
not
lying
to
you.
A
There
is
no
demo
namespace
in
cluster
Tegel,
so
it's
not
found
and
then
let's
restore
from
backup,
so
I'll
run
arc,
restore,
create
and
give
it
the
name
of
the
restore
that
I
like
to
call
it
and
I
tell
it
which
backup
to
create
the
restore
from.
So
this
takes
a
little
bit
of
time,
not
too
long,
because
this
there's
not
much
in
there,
but
we
can
go
ahead
and
check
and
see
in
cluster
two.
If
this
worked
so
we
have
deployments,
we
have
the
QWERTY
deployment.
A
We
have
the
service
that
was
restored
and
we
have
an
ingress
for
it
as
well.
All
right
so
we're
gonna
go
ahead
and
add
cluster
two
as
a
back-end
to
my
routing
cluster,
so
switch
back
to
the
router
and
I'll.
Do
our
add
back-end
cluster
too,
and
let's
see
what
happened
to
the
ingress
here.
So
what's
important
to
see,
let
me
scroll
up
just
a
little
bit.
We
have
some
new
annotations,
so
we
can
wait.
A
Each
back-end
in
the
cluster-
and
we
can
say
I-
want
a
hundred
percent
of
the
traffic
going
to
one
back-end
and
nothing
to
any
of
the
other
ones
or
you
can
divide
it
up.
However,
you'd,
like
one
thing
that
I
chose
to
do
for
this
demo,
is
I,
want
to
make
sure
that
the
addition
of
a
back
end
a
back
end
cluster,
isn't
something
that
automatically
puts
it
in
service
for
receiving
traffic.
A
So
you
can
see
that
cluster
two
has
a
default
weight
of
zero
and
it's
up
to
the
operator
me
in
this
case
to
decide
when
it's
time
to
bring
cluster
two
into
the
actual
routing
picture.
So
let's
give
it
ten
percent
of
traffic,
so
I'm
going
to
run
the
annotate
command
with
cube,
control
and
I'm,
just
changing
it
to
a
90/10
split.
And
if
we
take
a
look
at
the
ingress,
you
can
see
ninety
and
ten,
and
if
we
go
look
at
envoy,
we
can
take
a
look
and
see
envoy.
A
Has
this
concept
of
weighted
clusters,
where
you
can
you
can
send
different
amounts
of
traffic
to
different
backends,
and
so
here
I
have
ninety
percent
going
to
my
cluster
one
and
ten
percent
going
to
cluster
two.
So,
let's
see
if
it
actually
works,
I'm
gonna
run
the
same
load,
testing
script.
Again
it's
hitting
QWERTY
demo
and
if
everything
works,
this
will
send
about
ninety
percent
of
the
traffic
to
my
original
cluster
and
ten
to
the
second
one,
and
so
here
you
can
see
the
one
that
ends
in
dot.
A
Fifty
five
was
our
original
cluster
and
it
gets
the
majority
of
the
traffic.
If
I
go
into
the
web
browser,
we
can
take
a
look
and
you'll
notice
that
the
name
of
the
pod
here,
if
I
keep
refreshing,
does
eventually
change,
but
again
because
it's
the
90/10
split,
it's
going
to
hit
one
of
them
a
lot
more
than
it
hits
the
other
one.
A
Alright.
So
now
we're
happy
with
our
our
new
cluster
and
we
can
send
all
of
the
traffic
to
it
so
I'm
changing
the
weights
so
that
cluster
1
has
0
and
cluster
2
gets
100%
of
it,
and
we
can
double
check
the
ingress
to
make
sure
that
that
looks
good.
So
cluster
2
is
at
100.
We
can
take
a
look
at
envoy
configuration
I'll,
refresh
that
and
we'll
see
that
cluster
2
is
now
at
100
and
one
final
test
here.
A
A
A
A
So
when
you
look
at
this
picture,
this
type
of
setup
can
work
anywhere
doesn't
have
to
be
on
the
internet,
but
from
a
service
DNS
with
the
service
cluster
local
DNS.
That
works
within
a
single
cluster,
and
this
is
you
know,
cross
clusters.
One
of
the
things
I
did
look
at
when
I
was
putting
the
demo
together
was
maybe
trying
to
use
the
external
named
cname
support
that
you
have
with
services
and
I
just
didn't
get
far
enough
along
with
it
to
see
if
it
would
work,
but
it's
it
might
anyone
else.
Yes,.
A
A
Right
so
the
question
was,
if
you're
doing
get
operations
and
you
have
all
of
your
kubernetes
configurations
for
your
apps
and
whatnot
in
git
and
you
deploy
from
git
how
do
backups
fit
in.
So
when
we
were
working
on
Ark.
That's
a
question
that
comes
up
a
lot
and
the
way
that
we
look
at
it
is
get
ops
is
great.
A
But
if
you
need
to
tie
in
persistent
volume,
backups
it's
a
little
iffy,
so
Ark
can
help
you
associate
backups
with
your
existing
workloads
and
existing
things
that
are
running
in
kubernetes
and
what
we
would
like
to
get
to.
If
it's
not,
there
already
is
being
able
to
do
a
backup
and
restore
and
then
continue
to
do,
get
ops.
Beyond
that
point,.
A
A
Question
was:
would
it
be
ok
to
do
the
routing
cluster
only
when
you
need
to
migrate
and
then
get
rid
of
it
afterwards,
so
that
for
the
most
part,
you're
just
running
a
single
cluster,
yeah
I
guess
you
could?
It
would
just
depend
on
how
you
configure
your
ingress
traffic
from
the
internet
or
wherever
outside
the
cluster?
Is
you
know
if
it's
one
hop
or
two
you
potentially
could
do
that?
A
A
It
was
I,
think
I
heard
you
is
what's
the
advantage
of
having
a
routing
cluster
like
this,
instead
of
using
an
ingress
proxy
or
a
reverse
proxy
I
guess
it
depends
on
what
functionality
you
get
with
the
reverse
proxy
and
where
you
want
to
control
being
able
to
do
the
the
weighting
and
the
traffic
shifting
the
nice
thing
about
this
example.
Is
you
can
control
it
all
within
kubernetes
and
again,
depending
on
the
reverse
proxy
that
you're
using
you
may
not
have
the
same
sort
of
administrative
capabilities
from
within
the
cluster.
A
So
the
question
was
databases
such
as
Postgres.
How
do
we
back
those
up
so
arc
allows
you
to
run
custom
hooks
before
and
after
we
take
a
snapshot
of
a
volume
and
that's
essentially
a
queue
control
exec
into
your
pod.
So
if
you
need
to
freeze
the
file
system
and
then
take
a
snapshot
and
unfreeze
it,
you
can
do
something
like
that.
A
We've
also
been
considering,
but
we
haven't
made
much
progress,
the
concept
of
something
like
an
application
profile
or
a
you
know,
workload
profile
so
that
we
could
tell
you
could
tell
arc
this
pod
is
a
database
pod
and
it's
a
my
sequel,
pod
or
a
post-grad
pod,
and
then
we'd
have
preconceived
logic
for
doing
that
sort
of
backup,
but
we're
open
to
any
ideas
around
that.
It's
just
something.
We've
only
begun
thinking
about.