►
From YouTube: A Future Journey: How to Migrate 100 Clusters Between Clouds Without Downtime? - Tobias Schneck
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A Future Journey: How to Migrate 100 Clusters Between Clouds Without Downtime? - Tobias Schneck & Manuel Stößel, Kubermatic
Have you ever thought about migrating your Kubernetes clusters to another cloud provider to save costs? Yes? We too! Join us on an interactive journey to discover the main challenges of live migration at scale of etcd’s, traffic routing and application workloads from one cloud to another. The talk will discuss the current state of the technical concept, known problems and insides of the already proven migration steps for stateless workload. As part of the journey, we’ll see the differences between migrating one or one hundred clusters with productive workloads; What parts can be automated? What steps may need to be manual? Let’s see how an automated solution could look like in the future and what steps are missing.
https://sched.co/ekDU
A
Hello
and
welcome
to
our
talk:
how
to
migrate
100
clusters
between
clouds
without
downtime.
My
name
is
manuel
struss.
I'm
a
systems
architect
and
tech
lead
at
guru,
matic,
we're
doing
a
whole
bunch
of
kubernetes
and
cloud
native
consulting
and
we're
developing
the
kubernetes
kubernetes
platform
as
well
as
coupon.
A
A
A
There
are
also
some
technical
reasons
for
that.
So
again,
more
logistical
kind
of
reason
might
be
that
you
have
a
location
migration
of
a
data
center
or
you
might
want
to
migrate
to
another
network
segment
for
separation
of
concerns,
or
other
reasons
you
might
be.
Adapting
improvements
in
your
on-prem
and
cloud
environments.
Add
a
new
provider
that
you
want
to
use
like
new
features,
new
different
infrastructure
technologies.
A
Kubernetes
itself,
abstracts
infrastructure,
but
it
does
have
several
kind
of
dependencies
nonetheless
right,
so
it
does
consume
infrastructure
resources,
for
example
the
virtual
machines
where
the
cluster
itself
runs
on
it
uses
and
consumes.
The
network
provided
right.
The
ip
address,
space,
routing
and
firewalling
rules,
management
of
ingress
and
egress
traffic,
and
also
dns
as
always
and
external
storage
systems.
A
But
there
are
also
things
like:
storage
classes,
that
map
to
cloud
provider,
specific
storage
offerings,
and
sometimes
the
overlay
network.
You
use
has
some
dependency
on
a
cloud
provider
as
well
and
just
to
remind
ourselves
a
quick
overview
of
the
components
of
kubernetes,
the
central
one
being
the
api
server,
which
kind
of
handles
all
the
changes
in
state
of
the
cluster
itself
and
the
worker
nodes
with
the
cubelet,
which
runs
the
actual
workload
that
that
users
run
on
top
of
that
cluster
and
between
the
api
server
and
the
cubelet.
A
There
has
to
be
a
two-way
communication
happening
and
between
the
cubelets
as
well.
I
mean
between
the
worker
nodes
as
well,
and
that
is
one
main
challenges
where
we're
kind
of
solving
today
to
to
enable
this
ways
of
communication
and
that
directly
leads
us
to
our
actual
dependencies
when
we
migrate.
So
for
us,
the
application
workload
has
the
highest
priority,
but
we
need
to
ensure
for
that
to
be
the
case,
we
need
to
ensure
fundamental
networking
rules
that
kubernetes
expect
to
be
in
place.
A
Those
rules
are
that
all
containers
within
a
pod
can
communicate
unimpededly
on
layer,
4,
so
tcp
udp.
All
pods
can
communicate
with
all
other
pods
within
the
cluster
without
netting
and
all
nodes
can
communicate
with
all
parts
and
vice
versa,
also
without
netting
and
the
final
one,
the
ip
that
the
pod
sees
itself
is
the
same
ip.
That
others
see
the
part
as-
and
this
is
really
important
to
keep
up
so
that
the
actual
networking
between
parts
and
applications
works.
A
We
have
to
have
some
external
dependencies
that
need
to
be
reachable
right,
like
externally,
routed
ips
for
load,
balancers
and
notepad
services
within
the
cluster
and
dns
names
need
to
be
reachable,
resolvable,
right
and
storage.
Some
applications
might
have
to
have
state
that
needs
to
be
migrated
without
data
loss.
A
A
So
how
do
we
solve
for
that?
The
status
quo
is,
will
have
a
multi-cloud
setup
with
the
kubernetes
kubernetes
platform,
our
open
source,
kubernetes
management
platform,
and
it
has
a
concept
of
a
seed
cluster
that
holds
the
containerized
control
plane
of
the
user
clusters,
the
user
clusters
being
the
class
kubernetes
clusters
that
are
managed
by
kkp.
A
The
worker
nodes
itself
are
via
the
kubomatic
machine
controller,
which
is
a
cluster
api
conform
operator
that
translates
a
machine
deployment,
object
into
actual
machines,
vms
on
cloud
providers
and
we'll
use
canal
as
our
default
overlay
networking,
which
is
effectively
flannel.
The
exlan
overlay
networking
with
a
calico
network
policy
plugin
and
the
target
is
that
we
migrate,
the
user
and
c
cluster
control
planes
and
worker
nodes
to
a
different
cloud
provider.
A
We'll
keep
all
the
external
clustering
points
stable.
That
means
the
control,
plane,
kubernetes,
api
server,
endpoints
and
the
actual
application
endpoints
being
the
dns
and
ingress
routine
out
of
scope.
For
now
is
the
storage
replication.
The
assumption
is
that
the
actual
application
layer
manages
the
storage
replication
like
at
cd,
which
is
a
feature
that
we
will
use
to
migrate,
the
user
cluster
control
plane.
A
We'll
have
to
ensure
that
our
backup
and
recovery
procedure
for
the
seat
and
user
clusters,
but
also
for
the
application
workloads
works,
is
tested
and
and
proven
to
be
working
right.
We'll
should
create
a
target
cloud
cluster
as
a
reference,
in
our
case
an
aws
cluster
just
so
that
we
can
copy
and
paste
some
stuff
over
and
we'll
have
to
ensure
that
we
actually
control
the
dns
entries
and
be
able
to
to
switch
over
dns
entries
to
the
new
cloud
endpoints.
A
A
B
B
What
is
needed
to
ensure
the
traffic
is
reachable.
We
need
to
somehow
need
a
way
how
to
communicate
between
ports
and
ports
and
nodes
to
nodes.
For
that
we
create
a
vpn
overlay
by
a
daemon
set
and
route
and
traffic
of
the
cni,
in
our
case
kennel
through
the
vpn
network,
and
let's
ensure
that
we
have
maybe
different
network
segmentations,
but
still
can
talk
to
each
other.
B
If
we
have
this
client
to
client
vpn
traffic
enabled-
and
at
least
we
should
ensure
the
reachability,
that
means
that
we
try
as
long
as
possible,
to
keep
the
increase,
endpoint
stable
to
trans
and
then
transfer
the
workload
to
the
new
cloud
and
after
the
workload
is
the
new
cloud.
We
delete
the
connectivity,
okay,
how
sus
scan
looks
like
so
we
have
the
c
cluster.
What
host
in
the
containerize
control
plane?
Here
we
have
some
controllers
and
we
have
the
kubernetes
containerized
control
plane.
We
have
their
default
vpn
server.
B
Also,
we
have
a
machine
controller
which
is
configured
to
place
on
machines
on
the
vsphere
cloud
and
we
have
between
the
v3
workers.
We
have
an
overlay
what's
based
on
kennel
and
we
have
a
metal
ap
service.
What
creates
then
the
inbound
traffic
load
balancer
to
the
dedicated
application
ports
first
step?
How
to
migrate
is
now
to
deploy
a
vpn.
Daemon
said
this
vpn
daemon
set
ensures
that
we
have
an
vpn
client
at
every
worker
node.
B
So,
and
for
that,
we
need
to
pause
the
cluster,
because
our
cluster
is
controlled
by
a
cluster
controller,
and
this
cluster
controller
would
then
reconcile
the
machine
deployments
and
the
vpn
servers
so
to
ensure
that
this
does
not
happen.
We
pause
the
cluster
to
make
some
patches
there.
Next
step
is
after
we
have
the.
B
For
the
new
cloud
adopted,
we
can
update
the
cluster
spec
and
specificate
the
new
aws
cloud.
Then
the
machine
controller
gets
updated
and
we
get
a
new
machine
controller
instance
with
co
and
now
talk
to
aws
and
the
new
nodes
get
created
and
also
joins
our
vpn
network
and
the
join
so
also
the
kennel
overlay
routing.
B
The
cloud
controller
now
ensures
that
we
also
have
a
new
aws
lb.
What
get
created
chronolysis
llb
is
not
routed,
but
anyway
traffic
goes
from
here
to
over
the
metal
ap
service
to
the
dedicated
workload.
After
this
this
happens,
we
can
remove
the
workload
from
the
old
workers
and
move
the
workload
to
the
new
applications.
B
After
this
is
done.
We
can
also
re-name
the
dns
names
to
the
new
cloud
load
balancer
and
ensure
that
we
have
now
may
create
traffic
to
the
new
cloud
at
least
then
clean
up
the
old
resources
and
we
removed.
We
not
needed
any
more
vpn
overlay,
because
this
two
workers
can
now
talk
to
each
other
with
eth
interface
and
that's
how
we
migrated
it.
So
to
give
you
a
short
in
a
short
look,
what
already
is
happening?
B
We
create
some
demo
be
aware
that
currently,
this
project
is
like
not
fully
finished,
so
it's
more
a
proof
of
concept
state.
So,
as
we
see
here,
we
have
here
a
app
or
what
is
our
reference
echo
service,
and
this
is
deployed
on
the
cluster.
Let's
take
a
look
in
our
chiromatic
control
plane
and
in
this
control
plane.
We
have
the
dedicated
clusters
here.
So
first
we
have
here.
This
cube
called
migrate
cluster.
B
B
We
see
here
the
little
machines
running
and
this
one
we
want
now
to
move
to
aws
first
step.
What
we
already
did
is
that
we
deployed
a
vpn,
so
we
patched
our
vpn
server
and
for
that
we
see
that
we
have
a
cluster
spec
and
we
have
in
so
we
see
here
also
in
our
c
cluster,
that
everything
what
you
see
in
cubic
is
also
represented
as
a
cluster
crt
in
the
in
the
cluster
namespace.
B
B
B
B
Cluster
id
and
we
need
a
project
id
the
project
id
we
find
here
in
our
hubometic
url
here
to
approach
it.
Actually
we
can
copy
it
and
we
can
here
place
it.
So
what
is
the
first
step
we
create
in
backup
that's
the
backup
of
the
specification
of
this
cluster
so
yeah?
I
want
to
create
it
and
as
next
step
I
want
to
pause
the
cluster
and
yes
after
I
pause
the
cluster,
the
controller
does
not
care
anymore.
So
now
I
can
safely
update
my
cloud
provider.
B
B
B
So
let's
apply
that
and
see
what
happens.
So
we
see
now
we
have
configured
and
now
to
start
reconciling.
We
need
to
compose
the
cluster
that
the
cluster
controller
can
take
care
about
the
change
and
yeah.
Let's
see
what
now
is
happening,
so
we
have
now
the
cloud
spec
what
get
recreated
by
the
cloud
controller.
So,
hopefully
everything
works
well,
and
we
see
that
the
cluster
can
now
reconcile
yeah.
We
see
now
we're
getting
the
new
object
we
have.
The
vsphere
is
empty.
B
Okay,
this
looks
good
and
we
can
now
watch
that,
hopefully
yeah
we
see
the
api
server
is
reconciling
to
now
an
empty
cloud
provider:
okay,
cool.
That
was
the
first
step.
So
now,
as
next
step,
we
want
to
change
to
to
our
aws.
So
let's
go
out
of
this
view
and
let's
pause
the
cluster
again
like
currently,
depending
on
the
chromatic
controller.
We
need
this
two-step
upgrades
because
we
are
just
a
cube,
control,
client
and
not
operator,
and
and
now
we
can
patch
the
cloud
provider.
So
what
happens
now?
B
Yes,
I
want
to
patch
it.
Yes,
I'd
want
a
new
secret,
so
there
we
create
a
new
secret
reference
for
the
aws
credentials
and
then
we
created
a
new
cluster
patch
file
again.
So
let's
also
see
here
what's
different
now
so
we
see
now,
we
have
changed,
basically
removed
the
finalizers,
because
this
are
not
valid
anymore.
We
added
a
annotation
here
to
the
chromatic
avws
region.
We
patched
cloud
spec
aws
with
the
credentials
and
we
have
a
migration
a
new
data
center.
B
You
see
we
get
now
here,
also
aws
data
back
from
the
chromatic
controller
and
see
what
we
created
new
security
group
and
the
instance
profile
cool.
So,
let's
see
what
happens
with
my
components,
we're
now
seeing
yeah,
we
have
a
restarting
api
server
again.
It
started
now
with
the
new
club
prevent
credentials.
B
A
B
For
sure
I
need
to
write,
cube
config
and
now
we
should
go
to
the
api
server
of
our
cluster
and
see
that
here
we
have
specified-
hopefully
now
the
cloud
provider
this
here,
our
new
club
provider,
aws
and
now
that's
reconciling,
take
place
and
we
also
have
a
new
machine
controller
that
now
is
able
to
talk
with
aws.
So
what
we
can
now
create
is
the
new
aws
workers.
B
B
That
we
don't
upgrade
this
all
machines
and
yes
pause
done,
and
then
we
can
now
create
our
new
aws
workers.
Therefore,
we
need
first,
a
few
inputs,
so
here
we
need
to
specify
the
cluster
id
the
instance
profile
in
the
security
group,
but
chromatic
had
created
automatically
so
here
yeah.
I
want
to
see
the
metadatas.
B
Okay,
so
let's
create
that
one,
save
it
and
then
running
the
script
deploy
so
now,
hopefully,
fingers
crossed
we
grab
new
aws
workers.
So
yes,
I
want
to
create
one
here.
We
see
that's
now
rendered
in
in
the
machine
deployment.
We
see
our
security
group.
We
see
that
we
want
to
have
a
t3,
medium
and
yeah
that
in
the
usb
best
one
see
so,
okay,
then,
let's
deploy
it.
B
So
fingers
cross
yes,
as
has
been
created,
so
I
want
to
watch
the
creation.
Yes,
I
want
to
see
that
and
you
now
see
what
the
machine
deployment
we
have
a
new
machine
deployment
with
a
boot
target
aws
what
creates
two
new
machines
and
what
get
provision
now
in
atl?
Yes,
so,
let's
see
our
workload
is
still
running.
As
we
see
we
have
here
our
target
cluster
we
see.
Also
here
we
have
the
new
ubit,
would
retarget
aws
node
group.
B
B
A
B
So
we
have
created
new
machines.
So
let's
wait
until
the
points
say
I
get
booted.
In
the
meantime,
we
can
take
a
look
in
the
load
pencil.
So
hopefully
we
also
have
a
load
balancer
created.
That's
what
aws
cloud
controller
manager
will
create
because
we
already
have
a
service
type
load
balancer
in
the
cluster
and
that
aws
takes
over
and
creates
also
low
bands
on
their
side.
So,
let's
see
that's
not
the
right
one.
That's
the
right
one!
A
B
B
Yes,
I
want
to
connect
and
yeah.
We
see
here
that
we
have
here
a
flannel
route
and
we
will
soon
have
a
cube
or
a
interface
for
the
vpn
server
as
soon
as
that's
started.
Here
we
go
here.
We
see
that
interface
and
we
can
now
try
to
get
the
interconnection
between
one
cloud
to
another,
so
we,
I
then
connect
here
to
the
on-premise
node
on
the
down
and
let's
connect
to
that
one
ubuntu
and
test.
B
B
B
B
B
Good,
that's
corton
and
we
have
the
another
node
to
curtain.
So
that
means
that
the
node
should
be
now
marked
as
not
schedule
able,
yes,
so
scheduling
disabled
perfect.
So
we
can
now
use
the
cubecontrol
rollout
restart
feature
to
now
restart
our
deployment
of
the
echo
server
in
the
namespace
ico
server
to
trigger
like
a
row
in
the
release
of
the
echo
server
without
any
change
so
yeah,
let's
trigger
that
one
and
see
what's
happening
in
the
down,
we
see.
Okay,
that's
still.
B
B
A
B
B
Host
name:
we
see
that
the
yeah,
the
workers
are
running
in
a
new
cloud
and
is
still
reachable
through
our
old
endpoint
cool.
So
that
seems
to
work
like
now
that
the
next
step
would
be
to
migrate
all
other
workload
to
the
new
cloud,
remove
the
load
balancer
and
then
using
the
new
dns
name.
So
we
can
quickly
try
if
the
new
low
balance
is
now
listening,
so
yeah.
We
have
here
the
new
instances
and
maybe
the
dns
name
is
already
propagated
to
see.
B
B
So
how
are
the
next
step
looks
like?
So
that's
how
we
can
create
the
migration,
the
worker
user
clusters
and
to
move
completely
to
the
another
cloud
we
need
now
to
migrate,
also,
the
c
cluster
for
that
how
we
can
achieve.
That
is
the
same
way,
we're
reusing
the
same
principle.
So
here
we
we
have
the
workers
now
in
the
new
cloud,
and
now
we
need
to
make
migrate
control
plane.
B
How
does
this
look
like,
for
example,
here
we
have
our
seed
jns
for
our
own
api
server.
What's
projecting
these
remasters
there
we
have
our
now.
Our
aws
clusters
hosted
the
control
plane
and
we
have
the
workers
who
are
hosting
security,
voice
control,
plane
every
single
gcp,
then,
as
fast
first
step,
we
would
again
place
the
vpn
that
time
in
the
c
cluster
connecting
a
reworker,
node
and
masternodes
then
create
a
quorum
with
five
hcd
for
the
masternodes.
B
That
ensures
that
the
new
two
lcds
get
the
data
replicated
from
gcp
nodes
to
the
aws
nodes,
then,
as
next
step,
if
we
have
achieved
that,
we
can
remove
two
gcp
workers
and
still
have
the
quorum
of
three
so
three
or
five,
if
enough
is
enough
for
the
quorum,
and
we
can
therefore
also
change
now
the
the
the
dns
name
to
the
new
aws
club
provider
load
balancer,
so
there
now.
We
can
then
remove
the
quorum
also
from
five
to
three,
and
we
have
a
stable
control
plane
for
sure.
B
A
B
We
here
can
change
the
vpn
again
create
a
new
worker
nodes
and
then,
in
this
kind,
also
changing
here
at
the
cloud
provider
here
that
it's
pointing
now
to
aws
after
this
is
happened.
We
can
increase
here
in
chromatic
settings
the
default
value
of
entity
replicas
to
five.
That
means
that
the
programmatic
user
cluster
controller
manager
creates
two
new
entities
for
every
user
cluster.
That's
insurers
in
the
same
ways
as
me
migrated
the
lcd
on
the
top.
B
It's
migrating
also
for
the
for
the
users
clusters,
so
I
have
now
a
quorum
of
five.
It
is
running
but
still
pointing
to
the
old
dns,
but
between
clouds.
As
next
step,
we
now
creating
a
new
cloud
balance,
load,
balancer
and
rename
the
white
card.
So
that's
the
connection
from
the
user
cluster
workers.
What
we
already
migrated
to
the
new
cloud
load
balancer
and
yes,
then
we
switch
over
and
we
adding
one
new
aws
node,
replacing
all
gcp
nodes.
So
we
have
then
a
quorum
of
three
entities
per
user
cluster
on
the
new
cloud.
B
That
is
the
most
reasonable
target.
So
now
we
are
safe
and
we
can
remove
the
old
missingno
working
roads
clean
up
the
old
cloud
resources.
That
means
we
scale
down
to
lcd
level.
Three
we
remove,
then
the
vpn
overlay,
and
we
now
have
the
kernel
away.
What's
routing
between
the
workers
again
between
eth0.
B
So
then
we
have
successfully
migrated
every
user
cluster
into
seed
cluster
to
the
new
cloud.
So
that
is
the
approach.
What
we
think
are
is
possible
to
migrate,
really
100
clusters
without
downtime
and
in
a
scalable
way.
So
for
sure
we
are
not
right
now
that
we
just
press
a
button,
but
that's
our
future
target.
So
we
want
to
automate
also
the
clean
approach
procedure
and
for
sure,
if
you
want
to
really
make
it
scalable,
we
need
to
write
an
operator.
B
Currently
we
did
everything
by
a
yeah
hacky
help
bash
script
and
to
not
like
yeah
doing
that
by
hand
the
health
checks.
We
need
to
have
an
operator
who
check
the
house,
check
the
conditions
and
have
also
some
repair
options
and
retry
options,
but
therefore
the
do
we
need
this
reconsidering
pattern
match
in
the
same
way
as
is
match
for
our
cloud
migration,
where
the
chromatic
controller
creates
the
new
cloud
provider
and
so
on.
We
can
use
the
reconciling
also
for
the
migration
and
using
okay.
We
have
a
new
target
cloud
operator.
B
B
B
Okay,
one
more
detail
here.
Currently
we
tested
it
in
a
1.17
seat
cluster,
where
the
manage
fields
feature
is
not
included.
Somehow
this
makes
a
little
bit
of
trouble
between
okay,
you
have
an
like
patch
by
cube
control
and
you
have
an
operator
what
reconsidering
the
fields.
So
that's
why
currently
I'm
a
little
bit
limited
on
the
117
seed
clusters,
as
you
saw
in
the
demo,
the
user
cluster
can
be
on
118.