►
From YouTube: Vitess Intro: How to Migrate a MySQL Database to Vitess - Sugu Sougoumarane & Morgan Tocker
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Vitess Intro: How to Migrate a MySQL Database to Vitess - Sugu Sougoumarane. & Morgan Tocker, PlanetScale
Vitess is a database clustering system for horizontal scaling MySQL. This session will cover a high level overview of all the Vitess features, the architecture, and what database workloads are a good fit. We will then walk through a demo of live-migrating an existing MySQL installation into Vitess. Because Vitess also speaks the MySQL protocol, it is easy to retrofit scaling into your existing database systems.
https://sched.co/ZexD
A
So
hello
and
welcome
my
name
is
morgan.
We're
also
joined
by
sugu
for
answering
questions,
we're
both
maintainers
of
the
tests
suguru,
as
well
as
the
co-creator
of
the
test
and
founder
of
planet
scale.
Planet
scale
provides
commercial
services
for
the
tests,
so
today's
talk
is
an
intro
session
on
the
tests.
So,
let's
start
with
an
interim
vitesse
is
a
cloud
native
massively
scalable
database
to
your
applications.
It
looks
like
it's
mysql
and
actually
under
the
hood,
it
is
mysql
server
tests.
A
Orchestrates
mysql,
makes
it
highly
available,
makes
it
scale
horizontally
by
adding
new
servers.
This
is
a
feature
that's
obviously
not
available
in
in
standard
mysql,
and
really
allows
you
to
operate
it
in
a
cloud
native
way
and
it's
released
as
open
source.
It's
a
cncf
project.
So
just
some
quick
stats
about
it.
The
test
was
started
at
2010
at
youtube.
I
believe
it
went
to
production
in
in
2011.
A
B
A
A
So
it
started
at
youtube.
I
mentioned
that
fact,
but
now
the
test
is
used
in
many
places.
It's
used
by
slack.
It's
used
by
jd.com
jd
operates
the
largest
for
test
cluster
that
we
know
about,
which
is
thousands
of
tablets
we'll
get
to
that
terminology
in
a
second.
But
a
tablet
is
the
basic
unit
of
storage
within
the
tests
square
cash,
app
pinterest
hubspot.
You
know
many
of
these
names
you're
already
familiar
with
I'm
sure
for
developers
you're
familiar
with
github,
but
there
are.
There
are
several
other
others
as
well.
A
I
think
one
interesting
thing
to
point
out
while
we're
on
this
slide
is
that
the
adopters
are
also
maintainers
and
contributors.
So
if
we
highlight
on
the
slide
the
the
companies
that
are
employing
active,
maintainers,
slack
pinterest
hubspot
nozzle
square,
we
value
a
lot
from
their
contributions.
So
I
think
you
know
I
come
from
a
database
background.
There's
a
difference
between
you
know
when
a
product
is
created
by
a
product
company
and
when
it's
really
embraced
and
contributed
to
by
an
active
team
that
that
has
to
solve
real
world
problems.
A
So
you
know,
I
think
we
very
much
benefit
in
terms
of
making
sure
that
the
right
metrics
are
monitored,
there's
the
visibility
into
the
operating
of
a
system
and
the
right
bugs
are
fixed.
So
you
know
we
thank
these
companies
very
much
for
their
adoption
and
for
their
their
leadership
in
maintaining
the
tests.
A
So
I'm
going
to
flick
back
to
this.
This
slide
quite
a
bit
because
I
think
it
explains
a
lot
about
the
tess
of
a
test
is
a
distributed
system.
So
it's
got
more
moving
pieces
than
a
regular
mysql
and
there's
some
terminology
to
learn
to
really
benefit
in
understanding
the
tests
and
we'll
use
this
terminology.
When
we
get
to
really
the
the
meat
of
this
talk,
which
is
to
explain
how
we
would
migrate
from
mysql.
A
A
It's
a
little
bit
more
sophisticated
than
that
a
proxy
router
that
will
act
like
it's
a
mysql
server,
but
it
can
retrieve
the
data
from
multiple
mysql
servers
under
the
hood
to
be
able
to
you
know:
scatter
gather
or
re-piece
together
the
data
from
multiple
shards
or
or
segments
of
your
your
data
to
be
able
to
return
queries
to
your
application.
So
a
vt
gate
server
speaks
the
mysql
protocol.
A
You
speak
to
bt
gate
like
it's
a
mysql
server
with
a
mysql
connector
and
you
can
keep
using
the
same
connectors
in
your
application
that
you're
already
using
and
then
it's
routes
that
query
underneath
to
a
set
of
shards
which
are
going
to
be
using
mysql.
So
the
shout
outs
here,
they're
comprised
of
tablets.
This
is
the
basic
unit
of
data
that
we
have
in
in
the
tests
and
the
tablet
has
a
mysql
server.
You
can
use
mysql
as
made
by
the
mysql
team
at
oracle.
You
can
use
miri
db.
A
You
can
use
picona
server
in
a
different
mode
of
operation
which
we'll
get
to
as
well.
You
could
even
use
amazon
rds
or,
if
that's
what
you
would
like
to
use
and
the
the
shards
store
the
underlying
data
in
in
these
petitions
or
segments
of
data
that
that
the
tessa
recommends
you
keep
fairly
small
about
250
gigabytes
in
size.
So
it's
easy
to
be
able
to
scale
and
add
capacity
as
you
need
to
so.
The
state
is
stored
in
these
shards.
A
This
is
a
abstract
concept.
There's
no
technology
called
as
toppery
server
you
get
to
to
basically
take
your
pickup.
If
you
want
to
use
fcd,
that's
the
most
common
these
days
in
in
cloud
native
architectures,
but
console
and
zookeep
are
also
supported,
and
the
only
other
piece
that
I
haven't
described
on
this
slide
is
that
there's
a
management
gui
called
vtctld
which
can
make
those
changes
to
the
to
the
topology.
For
you
on
your
behalf.
But
it's
not
an
active
part
of
observing.
A
So
a
lot
of
detail
on
this
slide
and,
as
I
said,
I
will
flip
back
to
it
as
we're
describing
various
concepts.
But
for
now
I
want
to
summarize
it
as
vt
gate
is
the
mysql
proxy
that
routes
into
your
tablets,
which
is
where
you're,
storing
your
data,
and
there
is
a
bit
of
terminology
to
learn
to
really
understand
the
test.
A
A
We'll
get
a
little
bit
into
the
details.
We
talk
about
our
migration,
but
your
key
spaces
and
a
new
term
to
introduce
here
which
is
like
of
a
test
database.
They
can
be
either
sharded
or
uncharted
when
you
operate
in
uncharted
mode.
Basically,
almost
everything
is
supported
in
terms
of
sql
syntax.
If
you
operate
in
a
sharded
mode,
there
are
some
additional
restrictions
on
which
sql
is
supported.
A
Okay,
key
spaces
and
shots
key
space
is
it's
kind
of
like
mice.
What
mysql
calls
a
schemer
or
a
database
to
use.
That
term
would
be
a
little
bit
ambiguous,
though,
because
how
the
data
is
actually
stored
in
mysql
is
if
it's
sharded,
every
server
would
have
a
database.
The
test
uses
this
abstract
term.
That
makes
it
a
little
bit
easier
to
understand,
but
it's
like
a
logical
grouping
of
tables
and
tables
are
stored
in
shards,
shards
contain
replicas
or
or
multiple
tablets.
A
A
B
A
Small
shards
at
250
gigabytes
apiece.
You
know,
I
know
from
my
experience
of
operating
mysql
that
it's
well
with
a
well
optimized
system
fairly
possible
to
have
terabyte
size
database
instances,
but
the
reason
why
the
test
recommends
this
is
it's
very
easy
to
be
able
to
do
a
failover,
so
typically
of
a
a
shard
fails
or
a
server
fails.
A
A
A
Another
thing
for
operating
you
know
a
large
scale
system
is
that
you
want
to
make
sure
that
no
user
can
kind
of
impact
the
other
users
too
much.
So
you
do
get
some
natural,
fencing
or
basically
a
boundary
on
on
blast
radius,
if
you
do
pick
smaller
sizes
of
shards,
so
that
no
individual
user
can
kind
of
take
over
the
whole
system.
A
You
can
also
use
what
we
call
an
unmanaged
tablet
and
we
will
use
part
of
that
as
part
of
our
migration
strategy.
Where
you
can
say
the
test,
you
know
don't
manage
the
active
replication,
health
monitoring
and
things
like
that.
Just
treat
this
as
like
a
mysql
instance
to
route
to,
and
this
is
useful,
as
sort
of
like
an
entry
point
to
be
able
to
get
into
the
test.
A
A
A
Okay,
so
architecture
diagram,
we
have
vt
gates,
we
have
tablets
and
we
have,
as
I
said,
optionally
a
load
balancer,
a
couple
of
things
to
point
out
here
that
we'll
get
to
later
as
well.
By
having
these
extra
layers
of
the
load
balancer
and
the
vt
gate,
you
may
find
that
that,
although
the
test
has
very
similar
performance
characteristics
to
mysql,
because
it
is
mysql
when
it
executes
the
query,
it
does
add
additional
hops
to
be
able
to
get
to
mysql.
A
A
A
Does
my
application
tolerate
the
tess's
performance
characteristics
that
are
a
little
bit
different?
We
just
mentioned
that
and
then
the
third
one
is,
you
know
very
cautiously
in
operations
you
know.
No
software
is
perfect.
We
have
to
have
a
plan
to
be
able
to
back
out
if
we
can
and
actually
I'll
add
to
it.
We
have
to
have
ability
to
to
sort
of
test
it
in
a
safe
way
in
production
without
going.
You
know
completely
percent
of
the
test
on
on
day
one.
A
A
A
But
if
you
use
a
sharded
key
space,
actually
the
way
that
you
shod
your
tables,
which
you
specify
in
something
called
a
b
schema,
can
affect
which
queries
is
supported.
So
vt
explain,
can
accept
a
v
schema
and
a
schema
to
make
sure
that
it
validates
with
those
settings
that
this
list
of
queries
is
possible.
A
So
vt
explained
in
a
nutshell
here
I
say:
vt
explain
with
this
schema,
which
is
my
mysql
ddl,
and
this
v
schema
and
with
two
shards
will
this
sql
query
work,
and
then
I
will
get
some
information
to
say.
Yes,
here
is
how
it
works,
so
I'm
not
going
to
specifically
cover
the
sharding
in
this
talk,
but
I
think
it's
an
interesting
segue
to
very
briefly
touch
on
how
you
choose
to
shard
your
database.
You
know
often
you're
going
to
be
modeling
it
on
how
the
data
is
accessed.
A
So,
if
you're
a
sas
application,
you
may
be
shouting
by
the
tenant
to
say
that
for
this
individual
tenant
you
know
I
want
all
of
their
data
to
kind
of
reside
together
in
what
we
would
in
diabetes
terms,
say
clustering,
which
is
a
bit
of
an
ambiguous
term.
So
I
want
to
cluster
the
user
data
all
together,
so
they'll
get
very
efficient
queries.
A
How
you
do
that,
though
there
are
some
some
variations
where
that
might
be
a
good
idea,
or
it
might
be
very
bad
idea
if
you
tend
to
have
very
large
or
lopsided
distribution
of
tenants.
So
you
know,
let's
use
the
example
of
of
slack
where
individual
slacks
could
be
very
large
in
the
individual
tenants.
It
might
actually
be
better
to
shout
on
the
tenants
tenant,
so
the
individual
slack
channels,
rather
than
the
slack
deployment,
and
I
think
the
the
best
way
to
get
an
answer
for
this
is
to.
B
A
I
I
will
note,
though,
one
of
the
interesting
things
about
the
test
is.
It
does
make
it
very
easy
to
be
able
to
reshard
if
you've
designed,
if
you've
decided,
there's
a
better
way
to
layout
your
data
and
that
you
can
kind
of
change
the
scheme
and
reshod
to
a
different
way
of
laying
out
your
data.
You
know
the
alternatives
to
using
something
like
the
test.
Is
you
may
actually
change
your
applications
that
it
manages
the
shotting?
A
And
if
you
do
this,
I
think
it's
very
hard
to
kind
of
change
the
scheme
over
time.
Often
you
know,
if
you
don't
incorporate
this
into
your
design,
that
there
could
be
some
problem
so
interesting,
segue.
But
that's,
let's
look
back
to
to
our
migration,
so
we've
validated
that
our
queries,
work
with
bt,
explain
and
we
get
no
errors
and
it
looks
like
we're
ready
to
proceed
forward
or
almost
there
might
be
some
some
scenarios
that
are
hard
to
test.
A
Maybe
you
might
find
that
you've
tested
today's
queries
and
you
validated
them
and
there's
something
else,
that's
in
in
production,
and
you
know
we
have
to
make
sure
that
we
have
that
ability
to
to
back
out.
This
is
basically
just
the
first
level
lint
check
to
know
that
alpha
test
migration
is
going
to
be
successful.
A
A
A
All
of
this
is
sort
of
building
towards
the
confidence
to
make
sure
that
we
we
know
that
the
test
is
likely
going
to
be
a
bit
a
good
fit
for
our
application,
but
you
you
may
find
that
that
you
know
just
through
many
moving
pieces
that
that
there
will
still
be
some
queries
that
you
might
have
to
fix
this.
You
migrate.
A
A
If
we
use
the
load
balancer
as
well,
which
I
said
is
a
common
deployment
scenario
in
cloud
native
applications.
You've
got
two
more
hops,
so
you
know
the
the
scenario
where
this
is
going
to
hurt.
The
most
is,
if
we're
doing
a
lot
of
queries,
you
know
to
load
an
individual
web
page.
If
we
imagine
that
our
application
is
web
app.
So
there's
this
pattern
called
n
plus
one.
A
A
So
if
we
can
imagine
that
we've
got
a
hundred
queries
that
is
taking.
You
know:
100
milliseconds,
plus
a
little
bit
of
small
bit
on
top
once
we
add
two
more
network
hops,
we're
looking
at
300
to
maybe
more
milliseconds
for
that
that,
while
loop-
and
this
might
be
sufficiently
enough-
that
it
could
actually
impact
your
your
application,
I
tend
to
think
that
this
impacts
the
badly
design
applications
that
the
most.
A
If
you
can't
do
that,
let's
pretend,
for
example,
that
you've
got
a
while
loop,
where
the
results
of
the
previous
query
impact.
The
next
query
you
may
have
to
use
some
sort
of
parallelism
to
be
able
to
reduce
this
latency,
but
latency
is
something
that
you
should
keep
in
mind
in
your
evaluation
or
your
validation
of
the
tests.
It's
not
unique
to
the
test
by
any
stretch,
you
know
if
you're
evaluating
other
distributed
databases
as
well
as
the
tests.
A
A
So
if
you
can
just
deploy
to
five
percent
of
your
application
rather
than
a
hundred
percent,
this
is
generally
fairly
hard
to
do
with
stateful
services
because
they
start
modifying
data.
Often
it's
done
with.
You
know
deploying
new
versions
of
your
application,
but,
as
we
observed
when
we
look
at
that
architecture,
diagram
and
I'll
flip
it
back
up
in
a
second,
a
lot
of
the
tests
is
actually
stateless.
A
The
state
is
stored
in
mysql,
so
it
is
possible
to
deploy
the
tests
sort
of
like
on
the
side
and
route,
some
of
your
application
through
the
test
and
the
rest
of
it
directly
to
mysql,
to
be
able
to
do
that
that
validation,
and
if
we
have
problems,
we
can
just
roll
back
to
regular
mysql
as
is
required.
So
typically,
what
I'm
doing
when
I
first
start
out
with
five
percent,
is
a
man
I'm
operating
the
test
and
making
sure
that
it
it
reacts.
A
The
way
I
expect
it
to
so
is
the
cpu
load
acceptable.
You
know.
Maybe
it's
got
logging.
I
make
sure
that
it's
not
filling
up
disks
through
through
those
those
log
files.
I
look
for
errors
in
my
application.
I
look
for
errors
in
both
mysql
and
for
test
logs
and
I
start
ramping
up
that
at
low
to
eventually
getting
to
100.
So
let's
look
at
that
on
the
diagram.
First
I'll
throw
up
my
architecture
diagram
again
and
say
these
pieces
in
the
middle
vtg-gate
is
stateless.
A
The
topper
server
is
your
choice
of
ncd
or
zookeeper.
That's
where
the
state
is
stored,
server
protesters
managed
to
outsource
and
get
out
of
that
problem
of
storing
state
there
and
then
inside
of
the
shards
it's
using
mysql
to
be
able
to
store
that
that
state.
So
this
makes
the
test
inherently
having
the
building
blocks,
to
make
a
very
scalable
system,
you
can
just
add
vt
gates
to
add
capacity,
and
you
can
scale
by
adding
shots
to
add
capacity
in
that
stateful
service
as
well.
B
A
A
What
we
can
do
is
we
can
have
five
percent
of
our
application
service
or
application
speak
to
vtgate,
which
routes
to
vt
tablet,
which
uses
an
unmanaged
or
external
mysql,
and
this
is
where
I
would
sort
out.
So
there's
there's
a
few
ways
that
you
can
do
this.
Maybe
your
application
has
a
configuration
file
for
the
database.
Server
it
speaks
to.
You
may
have
to
change
whole
servers
to
to
go
to
an
individual,
vertex
or
mysql.
A
A
If
you
have
trouble
doing
this
as
well
in
your
your
app
because
you
don't
have
many
application
service,
what
you
could
also
do
is
you
could
put
a
sidecar
proxy
on
the
application
servers.
And
then
you
can
basically
pick
five
percent
of
of
connections
and
say
that
you're
out,
through
vt
gate
and
the
rest
go
to
existing
mysql
and
as.
A
A
So
our
goal
is
to
get
to
100
because
we
won't
be
able
to
take
advantage
of
the
tests
until
we
get
there
because
we
won't
be
able
to
shot
these
tables.
If
we
did
that
the
existing
applications
that
are
connecting
directly
to
mysql
are
just
going
to
be
really
confused,
but
we
should
find-
or
we
should
be
able
to
validate
in
a
production
way
that
you
know.
There's
no
added
latency,
there's
no
query
errors,
and
then
we
basically
just
keep
ramping
things
up
till
we
get
further
along.
A
Okay,
so
let's
talk
about
the
end
of
phase,
one
is
when
we
get
to
this
magical
goal
of
100,
where
we
have
all
applications
going
to
bt
gate
to
see
their
version
of
mysql
and
then
from
there.
What
I
would
typically
do
is.
I
would
recommend
that
you
move
from
using
your
existing
mysql
server
to
using
mysql
server,
that's
managed
by
the
tests,
so
we
have
now
a
tablet.
A
That's
that's
set
up
on
my
slide
where
we
have
a
mysql
server
with
that
bt
tablet
process,
and
then
I
want
to
what
I
want
to
be
able
to
do.
Is
it
move
the
data
from
my
existing
server
live
server
like
an
online
migration
from
my
existing
server
to
the
tests,
and
the
test
actually
supports
the
ability
to
copy
this
data
over
it
has
a
native
workflow
that
it
calls
remove
tables.
It's
part
of
a
feature
called
viewer
application.
A
A
This
is
what
we
call
a
switch,
so
we
have
move
tables
and
then
we
have
a
switch
to
be
able
to
cut
over
it
and
we,
like
the
term,
switch
because
it's
like
flicking
a
switch
that
final
operation
should
be.
You
know
very
safe
and
not
so
destructive
to
make
things
completely
live
just
a
few
requirements
on
the
source
to
be
able
to
do
this.
These
are
not
actually
the
defaults
in
older
versions
of
mysql.
They
are
the
defaults
in
mysql,
8
and
above,
which
is
a
little
bit
newer.
A
I
would
say
you
know
most
of
the
test
users
are
using
mysql57
just
for
a
little
back
there,
but
you
know
many
are
migrating
even
from
from
older
versions
and
it
it
works
pretty.
Well,
you
know
we
in
database
terms
use
the
the
terminology
online
to
say
that
it's
on
online
migration.
It
doesn't
affect
really
your
read
operations
or
your
write
operations
of
the
database,
it
just
copies
in
the
background,
while
operations
is
still
happening.
A
From
this
point,
we're
able
to
do
new
development,
basically,
we
can
start
charting
our
system.
We
cannot
start
adding
new
tablets
if
we
need
to
be
able
to
do
that.
We
can
manage
this
for
test
cluster
without
bt.
Ctl
gui
we're
completely
in
the
tests
at
this
point.
So
if
you
were
choosing
to
back
out
the
time
to
do
this
was
when
you're
at
that
five
percent
fifty
percent,
but
the
the
data
is
actually
still
in.
You
know
my
skill
formats
so
other
than
it
being
potentially
sharded.
A
If
you
do
wanna
change,
you
do
have
that
ability
for
test
doesn't
really
present
you
any
any
lock-in.
Okay.
So
our
setup
was
that
we
migrated
a
legacy.
Mysql
that
was
treated
as
one
uncharted.
Key
space
test
terminology
would
say
that
that
was
a
tablet
server
that
was
externally
managed,
and
our
new
setup
is
that
we
have
a
test
tablet,
that's
uncharted.
A
We
can
still
join
queries
between
tablets
in
each
key
space.
You
know,
even
when
we
move
to
sharded,
it's
recommended
to
keep
updates
contained
within
a
single
key
space,
just
in
the
tester's
design,
in
that
it
doesn't
want
you
to
do
a
two-phase
commit
to
be
able
to
do
updates.
So
we
didn't
talk
about
our
shouting
design
in
this
talk,
but
I
did
sort
of
allude
to
it
as
you
design
things
off
when
you're
designing
based
on
the
tenant
or
something
like
that.
A
If
you
want
to
learn
more
about
the
tests
and
particularly
about
this
workflow,
we
have
some
very
good
user
guides
on
the
test
website.
The
one
in
particular
that
I'm
basing
a
lot
of
this
talk
on
is
something
called
the
unmanaged
tablet
guide,
which
shows
how
to
set
up
the
tests
against
a
mysql
server.
A
B
B
B
There's
the
other
one,
which
is
actually
a
very
good
question
that
was
about
we
implemented
a
kubernetes
topo
server,
but
we
deprecated
helm
after
that,
and
we
did
not
port
that
implementation
to
the
operator.
It's
something
that
we
just
didn't
think
about
because,
but
it
is
less
important
about
which
topo
server
you
choose,
because
the
whole
idea
of
the
operator
is
that
it's
autopilot
it
should
just
run
for
yourself
run
by
itself.
So
you
shouldn't
have
to
worry
about
what
it
does
underneath
to
store
the
turpo.
B
But
if
there
are
use
cases
where
there
is
strong
reason
where
somebody
would
want
to
use
the
kubernetes
kubernetes
itself
as
a
toposerver,
it's
something
that
we
can
add
support
for.
But
this
is
the
first
time
I'm
hearing
this
question
and
that's
also
one
of
the
reasons
why
we
never
thought
about
it
because
it's
the
operator
works
so
well
right.
Now,
as
it
stands,.