►
From YouTube: Gone in 60 Minutes: Migrating 20 TB from AKS to GKE in an Hour with Vitess - Derek Perkins, Nozzle
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Gone in 60 Minutes: Migrating 20 TB from AKS to GKE in an Hour with Vitess - Derek Perkins, Nozzle
The holy grail of Cloud Native tech is to have zero vendor lock-in. That becomes extra challenging when dealing with stateful applications. By leveraging out of the box Kubernetes and Vitess features, Derek and his team were able to migrate a high throughput production workload of 20 TB from Azure (AKS) to Google (GKE) in under an hour. This workload consisted of dozens of services writing to MySQL, including heavy usage of the under-marketed pub/sub style message queue feature of Vitess. Derek will go into detail about the public Helm charts that were used to set up these workloads and how Kubernetes and Vitess were configured. We will also touch on a few ecosystem projects like external-dns, cert-manager that helped make the transition low-touch and seamless.
https://sched.co/UaYn
A
A
This
dream
that
someday
he's
going
to
get
it's
his
white
whale
that
he
just
can
never
do
and
that's
what
we
want
here
is:
how
do
we
avoid
lock-in?
You
know
it's
this
dream,
but
it
you
know
there
always
seems
to
be
one
thing
or
another
that
you
know
forces
you
to
stick
on
a
specific
cloud,
and
this
is
what
the
cloud
vendors
wants.
You
to
think.
A
It
also
made
me
think
of
Hotel
California
by
the
Eagles.
You
can
check
out
anytime,
you
like,
but
you
can
never
leave
if
I
had
a
walk-on
song.
That's
what
I
wanted
to
pick,
but
I
didn't
have
that
choice,
so
I
kind
of
want
to
talk
about
some
of
the
the
key
reasons
that
I
think
that
you
know
we
all
got
stuck
in
cloud
lock
in
you
know
provisioning
and
auto-scaling.
If
you're
using
a
specific
platform.
A
You
know
a
lot
of
people
are
using
RDS
or
whatever
database
platform,
because
they
are
scared
of
hosting
their
own
databases,
whatever
queue.
You
know
it's
kind
of
in
the
same
category
as
a
database.
How
do
we?
How
do
you
handle
persistent
storage
and
then
the
real
gotcha?
Is
this
network
egress?
You
know
they
let
all
this
data
in,
but
they
never
let
it
out.
A
Unless
you
pay
quite
the
price,
all
right
so
we're
you
know,
you've
picked
your
Heroku,
your
App
Engine,
your
lambda,
you
know
things
are
going
great
and
you
know
it's
great
when
you're
just
starting
up
you're,
saying
oh
I,
whether
you're
a
start-up
or
whether
you're,
just
you
know,
and
in
how
skunkworks
saying
you
know
we're
just
doing
a
proof-of-concept,
you
know
it's
I'm
sure
everyone
is
experienced.
You
know
this
proof
of
concept
that
sticks
around
for
a
decade
and
you
just
it's
so
hard
to
move
off
same
thing.
A
If
you
don't
have
you
know
a
DBA,
but
eventually
you're,
probably
going
to
hit
some
point
where
you
say:
oh,
if
I
could
only
enable
this
thing
or
that
thing
or
you
know
the
next
version
of
Postgres
or
my
sequel
launched,
and
you
know,
I
really
wish
I
could
use
X
feature,
but
it's
going
to
be.
You
know
two
or
three
years
before
that
hits
one
of
these
hosted
platforms
and
then
Q's
I
wanted
to
get
into
just
kind
of
real.
A
A
Similarly,
cloud
pub/sub
has
the
same
topic
and
subscriber
mentality,
except
it's
not
optional.
On
pub/sub
you,
everything
is
published
to
a
topic
that
has
zero
to
end
subscribers,
for
that
and
as
there's
the
only
one
I
haven't
actually
used,
but
they
have
their
service
bus,
that's
the
same
topic
to
fan
out
to
subscriber
mentality
and
then
finally,
we
kind
of
get
to
this
network
egress
that
can
get
real
expensive,
I'm,
not
actually
sure
I
didn't
make
this
chart
myself.
A
I
found
it
thanks
to
the
Internet
and
but
all
the
cloud
providers
have
roughly
kind
of
that
same
egress
costs.
So
let
me
get
going
to
a
little
bit
of
history
about
nozzle.
So
what
we
do
is
we
parse
Google,
we
like
to
say
we
when
I'm
talking
to
developers
I,
have
to
say
we
store
Google
on
Google,
so
we
parse
search
engine
results
and
then
we
provide
SEO
related
analytics
to
big
customers
that
are
looking
to
understand
who's,
ranking
where
why
they
might
be
ranking
and
as
much
metadata
as
we
can
provide
about
it.
A
So
from
the
beginning,
like
I
kind
of
mentioned,
we
did
a
proof-of-concept
on
App
Engine,
we're
still
in
App
Engine
six
years
later,
and
then
this
logo
here
at
the
bottom,
is
cloud
tasks
which
we
actually
have
a
it's
been
amazingly
stable.
So
there
are
definitely
reasons
that
you
know
choosing
some
of
these
hosted
products
are
great,
but
in
our
actual
codebase
the
error,
for
if
a
message
fails
to
add
to
cloud
tasks,
it
would
just
omits.
A
All
three
of
these
do
the
other
kind
of
cloud
providers
digitalocean
provide
some
as
well,
but
that
Azure
number
I
don't
think
it
actually
is
the
case
anymore,
but
they're
providing
this
much.
But
this
is
my
what
about
Bob
reference
that
as
we
just
swooped
in
with
their
bagful
of
money,
and
they
tried
to
buy
my
loyalty,
we
had
been
on
App
Engine,
but
all
of
a
sudden,
Oh
360,000.
A
If
it
wasn't
obvious
from
the
title
to
talk,
it
was
not
so
we
started
using
aks.
This
is
a
pretty
good
representation
of
what
it
felt
like
when
we
started
it
was
a
beta
product,
and
so
that
was
to
be
expected,
but
then,
for
some
reason
they
said.
Oh,
it's
GA.
Now,
despite
it
being
exactly
the
same,
some
specific
things
that
made
us
decide
to
leave.
Aks
there
was
a
hundred
node
limit
which
may
or
may
not
still
exist
to
this
day.
The
docks
aren't
consistent
on
that.
A
We
actually
had
a
2-day
outage
while
it
was
in
GA,
because
they
just
artificially
throttled
the
CPU
on
our
masters,
and
it
took
two
days
for
him
to
figure
that
out,
even
with
a
support.
You
know
the
pro
support
contract,
so
that
was
awesome
and
then
finally
kind
of
the
nail
last
nail
in
the
coffin
was
the
pricing.
A
If
you
didn't,
you
know,
lock
yourself
in
was
significantly
more
and
with
that
node
pool
a
lot
of
most
of
our
workloads
that
we
run
our
batch
workloads
and
so
the
ability
to
just
use
a
preemptable,
node
pool
and
cut
our
cost.
Seventy
percent
and
it's
a
spot
instance
basically
for
the
AWS
users
in
the
room.
You
know
really
made
a
huge
impact
on
our
cost
without
us
having
to
commit
to
anything.
A
A
Actually,
we
are
still
on
our
credits,
so
I
didn't
care
about
egress.
To
be
perfectly
honest,
but
that's
only
if
you
migrated
out
before
your
credits.
Expire
I
gave
myself
an
hour,
so
it's
I
was
kind
of
forced
to
be
gone
in
an
hour,
so
kubernetes
is
obviously
great,
but
I
kind
of
want
to
talk
about
Vitesse
and
exactly
how
its
architecture
lends
itself
to
this
way
of
treating
your
data
and
handling
it
in
a
cloud
agnostic
way.
So
you
start
off.
You
have
all
of
your
apps
that
are
running
they're.
A
You
know
expected
to
point
at
a
database
which
is
exactly
how
the
architecture
works.
You
pointed
at
a
load
balancer
in
kubernetes,
it's
just
a
service,
and
then
it
points
at
this
VT
gate
layer.
That
is
a
stateless
component
that
will
auto
scale
up
and
down.
That's
what
presents
itself
to
your
application
as
the
actual
my
sequel
interface.
A
It
also
has
a
G
RPC
interface
for
that,
but
most
people
will
just
want
to
point
at
that
my
sequel
protocol
and
then
once
you
hit
this
VT
gate
layer,
that's
where
it
makes
some
of
these
decisions
about
what
shards
specifically
to
look
at.
If
you
zoom
in
here
on
one
of
these
shards,
there's
kind
of
this
blue
and
orange,
so
VT
tablet
is
a
component
that
runs
generally
in
the
same
pod
as
your
actual
my
sequel,
D
process
and
is
what
actually
communicates.
A
So
it's
a
G
RPC
internally
from
VT
gate
to
the
VT
tablet,
which
then
generally
communicates
over
a
localhost
UNIX
socket
into
my
sequel.
So
you
get
pretty
low
latency
there.
If
you
do
work
at
a
company
that
feels
like
no
I
have
to
put
this
data
in
a
roar
or
cloud
sequel,
because
kubernetes
just
can't
handle
stateful
workloads.
It
can't
also
do
that,
but
for
kind
of
the
reasons
I've
said,
I
you
really
like
sugu
said
in
his
keynote.
A
That's
not
all
that
different
from
the
kubernetes
api
server
and
then
a
VT
control,
D
that
communicates
with
that
and
lets.
You
run
the
test
related
commands
on
a
sidenote
one
of
the
original
contributors
to
Vitesse,
who
then
moved
to
kubernetes.
He
actually
snuck
a
commit
in
that
and
I
think
it
was
more.
The
1:9
release
notes
that
it
is
not
cube
cuddle
and
he
gave
the
canonical
way
to
pronounce
cube
control.
So
it's
in
the
docs,
and
so
it's
Canon
I'd
prefer
to
not
hear
cube
cuddle
ever
again.
A
A
So
Vitesse
has
the
concept
internally,
that
are
it's
called
cells
that
are
going
to
be,
you
know
roughly
equivalent
to
an
AZ,
but
it's
just
a
whatever
your
data
center
or
your
your
kind
of
your
blast
radius
that
you
want
it
to
be,
and
then
it
has
a
global
topology
server
and
actually,
let
me
jump
back
here.
Real
quick
you'll
notice
that
this
topology
server
is
not
in
the
actual
request
flow
as
queries
come
in
and
that's
very
purposeful,
because
it's
built
to
scale
now
like
subbu
mentioned
in
the
keynote
JD
comm,
is
hitting
thirty.
A
A
So
we
had
a
choice
about
whether
we
wanted
to
try
and
do
a
zero
downtime
movement,
which
is,
would
be
full
supported
in
this
scenario
by
simply
reparent
omastar
over
2g
ke,
and
then
all
rights
would
be
directed
there
again.
Assuming
you
get
the
multi
clustered
networking
taken
care
of,
and
then
you
just
turn
down
the
azure
region
and
your
apps
don't
ever
know
the
difference.
So
it's
pretty
fantastic
that
way
and
as
you're
doing
those
fail.
A
Overs,
like
I,
mentioned
it's
generally
a
zero
downtime
reparent,
so
the
database
running
in
kubernetes
is
handled,
but
this
next
one
is
a
feature.
That's
near
and
dear
to
my
heart.
It's
the
unsung
hero
that
is
Vitesse
messages
that
I
begged
into
existence
about
two
and
a
half
years
ago,
because
I
thought
I
was
going
to
migrate.
Much
sooner
to
Vitesse
in
kubernetes,
I
was
actually
while
HubSpot
was
at
one
of
the
talks.
A
They
talked
about
how
HubSpot
was
running
ten
terabytes
of
general
log
to
try
and
improve
sequel
compatibility,
but
apparently
I'm
a
great
beggar
and
I
got
Sugo
to
attack
on
this
much
more
fun
to
code.
The
test
messages
that's
much
more
fun
than
increasing
sequel
compatibility,
so
it's
the
best.
So
all
that
this
queue
is
so
it's
functionally
pretty
similar
to
the
queues
that
we
looked
at
before
there's
a
topic
that
will
can
fan
out
to
subscriptions.
A
So
here
again
it's
just
a
my
sequel
table
under
the
hood,
which
provides
a
lot
of
benefits.
You
just
make
this
specific
table.
This
is
an
event
stream
message
table
and
then
has
a
lot
of
the
fields
that
you'd
expect
what
time
was
as
scheduled.
What's
the
ID
the
next
time
it's
going
to
run,
you
know
how
many
times
has
it
retried
and
then
you
can
actually
add
on
this
key
space.
Id
attributes
data,
that's
what
we
use
internally,
but
you
can
actually
define
whatever
fields
you
want.
A
A
Depending
on
how
you
feel
about
foreign
keys,
I
love
them,
they
make
operation
a
little
bit
harder,
but
it
makes
me
feel
better
and
less
angry
at
my
developers
and
so
warms
my
heart.
So
you
can
use
foreign
keys
just
like
you
would
anywhere
else
and
then
down
here
in
the
comment
at
the
bottom.
This
is
where
you
define
all
of
the
actual
cue
attributes
that
you
need
to
use.
So
the
topic
here
points
at
the
topic
that
it's
subscribed
to
the
Vitesse
message
lets.
A
You
know
that
this
is
a
message
table
and
has
received
special
treatment
by
the
Vitesse
message
manager
and
then
some
your
act
timeouts.
How
long
you
want
your
attention
to
be,
and
some
other
things.
So
it's
very
simple
to
set
up
and
again
it's
a
very
simple,
my
sequel
world
you're
living
in
which
is
completely
vendor.
Agnostic
and
I
see
a
lot
of
people.
You
know
they
kind
of
reflexively
want
to
reach
for
Kafka
or
reach
for
gnats,
or
something
that's
kind
of
a
dedicated
message.
A
And
now
you
have
to
have
someone
that
understands
that
or
if
you're,
like
some
organizations,
you
just
deploy
data
sources
willy-nilly
and
you
just
hope
that
they
don't
break
because
you
don't
have
any
internal
expertise,
but
by
pushing
it
through
the
tests,
you
can
say:
okay,
we
know
we
understand
my
sequel
and
we
know
how
to
make
that
scale
and
by
doing
that,
it's
amazing
when
you
can
rely
on
a
database
transaction.
To
both
add
data
to
the
next
cue.
Do
whatever
data
work.
A
You
need
to
be
done
and
then
act
that
message
back
to
the
cue
and
it
eliminates
a
whole
class
of
errors
that
that
could
pop
up
from
you
have
an
orphaned
record
because
either
it
got
add
it
to
the
queue
and
the
ACK
failed.
Or
you
did
some
data
work,
but
then
the
queue
went
down
by
relying
on
my
sequel
to
enforce
that
transaction.
You
know
that
you
just
don't
even
have
to
write
cleanup
code
for
all
those
orphaned
records.
A
Another
problem:
you
know,
with
queues
that
we've
seen
in
the
past,
it's
hard
to
see
what
you
have
in
your
queue,
it's
hard
to
modify
it
in
case
something
got
added,
and
it's
just
sequel.
You
can
wreak
you
something
with
a
simple
update
statement.
You
can
you
don't
have
to
write
some
code
to
loop
through
and
see
what
data
you
have
in
there?
A
So
now
I
wanted
to
dive
into
okay.
Now
that
we
kind
of
have
this
background
on
the
tests,
you
know,
how
did
we
use
that
to
get
back
to
GK?
So,
like
I
mentioned,
we
have
these
two
options.
We
could
have
figured
out
the
cross
cluster
networking
done
a
zero
downtime
like
I
mentioned,
most
of
and
actually
all
of
the
workloads.
A
At
that
time
we
were
running
we're
a
batch
and
so
taking
a
little
bit
of
downtime
was
an
okay
choice
for
us
and
seemed
a
lot
simpler
than
trying
to
figure
out
this
cross
cluster
networking
when
it
was
just
going
to
be
a
temporary
solution,
I'm
also
not
a
network
guy,
so
we
decided.
Okay,
all
we
have
to
do
is
take
a
backup
on
a
KS
turn
things
down
and
restore
on
GK,
so
to
make
sure
that
we
minimized
that
downtime
as
much
as
possible.
A
A
A
We
did
it
a
few
times
and
we
were
struggled
to
get
that
back
up
taken
transferred
over
and
then
boot.
It
up
and
running
have
a
running
my
sequel
instance
on
the
other
side
in
less
than
seven
days,
mostly
because
it
just
takes
time
to
you,
know,
work
at
these
big
chunks,
where
Vitesse
has
a
philosophy
of
250
gigabyte,
shards,
give
or
take
it's,
not
a
technical
limitation,
but
it
means
that
you
can
generally
backup
and
restore
within
about
15
minutes.
A
Since
we
were
crossing
clouds
we
gave,
we
knew
we
were
probably
to
take
a
little
bit
longer
and
so
the
it
was
night
and
day
the
whole
transition
experience.
We
also
shut
down
external
DNS
for
those
of
you
haven't
used
it.
You
just
set
up
some
ingress
annotations
external
DNS,
watches
those
and
they
integrate
with
whatever
DNS
provider
you're
using
and
they
point
you
at
whatever
the
load
bouncer
IP
address
is
for
the
load.
Balancer
the
nginx
ingress
so
we're
coming
through,
since
we
were
deploying
it
on
the
GK
Eastside.
A
We
turned
it
down
here
so
that
we
didn't
have
a
race
condition
of
them
both
trying
to
handle
those
same
DNS
entries
and
if
you
aren't
trying
to,
if
you
don't
have
any
external
consumers
and
you're
okay,
switching,
you
know
your
DNS
entries,
it
does
have
the
ability
to
you
can
kind
of
set
a
prefix.
So
you
can
say
okay.
This
is
my
aks
external
DNS
manager
and
my
gke,
and
that
keeps
them
from
clobbering
each
other.
A
A
So
then,
on
the
gke
side,
we
had
to
bootstrap
things
like
I
mentioned
I'm,
not
in
that
CD
expert
and
I
thought
to
myself.
I
wonder
it
if
I
can
just
pretend
that
that
didn't
exist
and
just
let
the
test
bootstrap
itself,
it's
pretty
good
at
it,
and
so
I
first
tried
I
didn't
actually
even
take
a
backup
or
move
that
global
topology
server.
So
what
I
did
is
I
deployed
after
the
that
backup
had
finished,
we
deployed
the
Vitesse
helmet
art,
but
only
with
one
replicas
per
tablet.
A
That
makes
it
easier
for
you'll
see
here
later
now.
We
run
this
tablet
externally
repented
to
choose
the
master
for
every
single
shard
and
that
process
itself
bootstrap
to
the
global
topology
server.
And
then,
when
we
scaled
up
the
replicas,
they
were
able
to
query
that
topology
server
and
then
auto
joined
and
the
test
set
up
all
the
replication
for
those
new
replicas
back
to
the
master,
and
you
know
when
that
was
done.
I
kind
of
thought
to
myself.
A
It
just
worked
it's
That
moment
when
you
kind
of
you
were
prepping
for
like
this
long
battle,
and
then
it
didn't
happen,
and
it
just
feels
too
easy
and
I
was
just
thinking
that
something
was
gonna
go
wrong.
The
only
thing
that
did
go
wrong
was
also
side.
Note:
I
saw
a
service
when
I
was
actually
deploying
that
home
chart,
but
I
was
like
I
wrote,
the
helm
chart
and
I
was
like
I.
Don't
know
why
I
put
this
in
there,
I
probably
don't
need
it
so
I
took
it
out
and
it
didn't
work.
A
A
Honorable
mentions
to
the
ingress
controller
cert
manager,
an
external
DNS,
for
make
that
you
know
not
having
to
manage
all
those
certs.
Everything
happens
over
SSL
happened
through
those
same
DNS
entries
and
on
Google
cloud.
Most
of
those
provisioning
steps
for
these
happened
within
seconds
two
minutes
compared
to
the
you
know,
tens
of
minutes
two
hours
that
we
sometimes
saw
in
aks.
A
So
some
future.
You
know
thoughts
like
if
you
did
want
to
run
multi
cloud
or
even
multi
region.
You
know
again,
as
far
as
the
test
is
concerned.
It
doesn't
particularly
care
if
you're
in
a
separate
region
or
you're
in
a
different
cloud.
As
long
as
your
networking
is
set
up
and
all
the
components
can
talk
to
each
other,
so
there's
a
couple
of
cool
things.
A
There
was
a
talk
before
given
by
g10
in
deep
fee
about
GDP
our
concerns,
and
you
can
actually
set
up
your
Vitesse
charting
strategy
to
be
country
or
jurisdiction,
aware,
and
that
provides
some
pretty
cool
opportunities
to
lower
your
Layton
sees
so
that
your
clusters
will
all
talk
to
their
local
masters.
So
you're
not
waiting
for
those
reads
or
writes
to
go
across
the
globe
back
to
wherever
you're
kind
of
home
cluster
is.
But
you
because
my
sequel
does
asynchronous
replication.
A
A
Cockroach
DB
actually
has
an
interesting,
follow
the
workload
feature
where
you
kind
of
watch,
where
the
bulk
of
your
rights
and
it
kind
of
generally
follows
the
Sun
as
you
go
through
business
hours
throughout
the
world,
so
it'd
be
pretty
trivial
to
then
kind
of
move
your
master
around
between
these
multi
cloud
or
multi
region
clusters,
so
that
the
bulk
of
your
operations,
the
master,
is
closest
to
wherever
those
are
happening.
So,
anyhow
thanks
everyone
for
coming.
It
was
a
blast
like
I
said
it
was
almost
scarily
easy
to
move
across
clouds.
C
C
A
D
A
A
A
A
I
just
copied
so
the
backup
the
test
backups
they
by
default
right,
there's
an
integration
to
send
it
to
GCS
or
to
s3,
and
so
it
was
I
think
we
have
50
shards
at
the
time
that
we're
all
independent
Lee
writing
to
GCSE.
So
I,
don't
know
what
those
exact
limitations
are,
but
it
was
big
enough.
Okay,
all.