►
From YouTube: Zero Downtime Data Relocation with Vitess - Liz van Dijk & Deepthi Sigireddi, PlanetScale
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Zero Downtime Data Relocation with Vitess - Liz van Dijk & Deepthi Sigireddi, PlanetScale
Vitess has a flexible sharding architecture and natively supports "cells" which correspond to infrastructure located in multiple locations. This allows for the creation of jurisdiction-aware database clusters that solve data locality without having to re-architect your application. Also, Vitess's built-in support for resharding workflows makes migrating from existing databases into databases resident in multiple locations easy. In this talk the speakers will show how to build a custom sharding scheme in Vitess that respects data locality requirements. They will then demonstrate a database cluster built using this scheme that enables transfer of existing data belonging to people from 8 different countries from one jurisdiction to four(4) separate jurisdictions with zero downtime at the application level.
https://sched.co/ZeoD
A
B
And
my
name
is
liz
van
dyke,
I'm
a
solution
architect
with
planet
scale.
I
joined
the
company
about
nine
months
ago,
but
given
everything
there
is
to
know
about
the
tests
and
kubernetes,
I
still
feel
like
a
newbie
most
of
the
time.
My
background
is
mainly
in
mysql
since
I
previously
worked
at
a
company
called
racona
for
about
seven
years
and
it's
been
really
exciting
to
branch
out
into
what
I
consider
to
be
the
logical
evolution
of
mysql
in
the
kubernetes-based
world.
B
So
it's
not
the
first
time
we
talk
about
data
locality,
but
it
does
feel
as
though
every
time
we
get
to
share
this
topic,
it
becomes
more
relevant.
We
are
recording
this
just
weeks
after
the
european
court
of
justice
repealed
the
usc
privacy
shield
agreement,
particularly
global
hr
platforms,
have
needed
to
pick
up
on
this.
B
And,
while
that's
certainly
true,
this
is
a
trend.
That's
not
reversing
anytime
soon.
So
after
the
ruckus
of
getting
our
companies
in
good
shape
for
gdpr
back
in
2018,
business
leaders
have
been
rightfully
aiming
to
get
ahead
of
whatever
new
requirements
are
coming
and
are
considering
what
it's
going
to
take
to
reshape
those
data
stores.
B
So
we
have
a
lot
going
on
on
this
diagram.
The
quickest
way
to
explain
what
the
test
is
is
as
a
middleware
layer
between
your
application
and
multiple
clusters
of
mysql
servers
to
your
application
on
the
left
of
the
dotted
line,
it
presents
itself
as
a
single
giant,
mysql
database.
It
speaks
to
sql
protocol.
B
You
use
standard,
mysql
connectors
to
communicate
with
it,
and
it's
pretty
straightforward,
of
course.
Internally,
there's
a
lot
more
complexity.
The
test
is
built
to
accommodate
transparent,
horizontal
sharding
and
there's
many
components
that
contribute
to
that.
Just
in
case
the
concept
of
horizontal
sharding
is
not
yet
100
clear.
The
basic
principle
behind
it
is
that
we
take
the
data
within
a
single
given
table
and
we
spread
it
across
multiple
shard
clusters.
B
So,
for
example,
if
you
have
a
table
that
contains
four
billion
rows,
we
can
spread
those
evenly
across
four
shards
containing
1
billion
rows.
Each
the
schema
for
each
of
these
shards
is
identical
and
to
your
application.
The
data
still
looks
as
though
it's
coming
from
one
and
the
same
table
in
this
talk.
We're
going
to
be
focusing
primarily
on
what's
in
the
query
path,
which
is
represented
by
the
white
line
sphere
outside
of
that
list,
btctld
and
our
topology
servers
those
help
us
manage
the
infrastructure
but
they're
kind
of
less
relevant.
B
To
our
talk,
I
do
want
to
draw
some
special
attention
to
vt
gates
by
itself.
Vtgate
is
your
application's
entry
point
to
the
cluster,
it's
a
very
light,
stateless
proxy,
which
contains
an
sql
parser,
and
it
keeps
itself
informed
as
to
the
state
of
the
cluster.
So
it
knows
exactly
how
to
break
down
your
requests
at
any.
Given
time
it's
going
to
transparently
select
the
correct
shard
for
you
to
select
from,
and
it
supports
and
respects
a
very
large
variety
of
native
sql
terms
like
joins
and
transactions.
B
B
B
The
v-schema
is
important
for
us
to
understand
because
it's
where
we
define
our
application,
specific
sharding
needs,
as
well
as
where
we
collect
configurations
for
any
of
the
test.
Specific
features
we'd
like
to
use
on
a
table
by
table
basis
so
effectively.
The
v-schema
is
where
you
build
the
sharding
logic
for
your
application,
allowing
it
to
remain
abstracted
from
your
code.
B
Now,
at
its
most
basic
definition,
charting
of
tables
happens
by
assigning
every
row
a
specific
key
space
id.
This
uses
a
computed
column,
the
so-called
sharding
key,
and
it
will
associate
each
row
to
a
physical
address
in
a
very
specific
chart,
and
that
means
that
each
shard
is
defined
as
the
owner
of
a
specific
range
of
key
space
ids.
B
B
It
connects
to
any
existing
version
of
a
running
mysql
server
that
can
be
any
flavor
running
anywhere.
We
also
include
cloud
instances
for
that,
like
rds
or
aurora
and
effectively
what
it
does
is
it
adopts
that
instance
into
the
greater
the
test
cluster
and
make
sure
that
it's
got
a
correct
copy
of
the
data
and
is
configured
to
execute
its
given
role.
B
They
allow
us
to
spread
a
given
key
space
across
multiple
geographical
locations.
The
actual
definition
of
a
cell
doesn't
need
to
imply
geographical
spread
and
we
could
define
them
as
different
racks
in
the
same
data
center.
But
for
the
purposes
of
our
demo,
we're
opting
to
let
the
cells
be
defined
as
the
geographical
boundaries
for
the
data
set
forth
by
the
data
locality
requirements.
B
Now,
to
accomplish
a
truly
transparent,
your
application
distribution
of
data,
a
normal
horizontal
sharding
strategy
is
not
going
to
suffice.
A
strategy
that
evenly
spreads
your
data
across
the
shards
we're
going
to
need
to
build
in
some
additional
logic.
That's
specific
to
the
rule,
sets
that
we're
trying
to
comply
with,
and
we
want
to
keep
in
mind
that
this
is
a
moving
target
because,
as
we've
seen
in
the
past
few
years,
those
rules
can
be
very
dynamic
and
they
change
rather
quickly.
B
B
So
a
list,
a
list
of
the
cells
across
which
will
establish
our
space
deeply,
is
going
to
show
you
how
to
establish
a
sharding
function
that
allows
us
to
confine
location-specific
rows
to
the
correct
charts
and
she'll.
Show
you
how
to
execute
that
in
place,
beginning
from
an
existing
uncharted
data
set.
A
The
sharding
scheme
we
will
be
using
for
the
demo
is
to
use
a
region
json
windex.
This
is
a
windex
that
is
a
multi-column
index
until
late
last
year,
with
us
only
had
a
single
column
indexes,
but
as
as
we
realized
that
we
should
be
working
towards
data
locality
requirements,
we
added
a
multi-column
index
and
what
this
specific
region
json
of
index
implementation
does.
A
A
A
We
can
look
at
the
instances
that
we
have
in
this
cluster
and
filter
them
by
shard.
So
we
have
three
uncharted
instances:
one
in
asia,
one
in
europe,
one
in
the
us
and
in
each
chart
we
have
three
instances,
but
they
are
all
located
in
the
same
region.
So
for
dash
40,
we
have
us
40
to
80.
All
of
the
instances
are
in
europe
and
80.
All
of
the
instances
are
in
asia.
A
A
Let's
now
take
a
a
look
at
the
v
schema
that
we
have
created,
so
this
is
a
resharding,
so
we
have
created
a
v
schema.
That
says,
sharing
is
true
and
there
is
one
windex
which
is
of
type
region
json
and
we
are
calling
it
region
vdx
and
we
are
going
to
give
it
a
country
to
byte
map
file
which
contains
the
json,
and
this
windex
uses
one
byte
to
represent
the
region
and
on
our
customer
table.
A
A
A
A
A
A
A
So
we
see
that
the
data
has
all
been
copied
over
and
we
can
access
this
data
by
routing
our
queries
to
a
replica.
The
next
step
is
to
switch
rights.
So
let's
do
that
now
before
we
switch
rights.
Let
us
start
a
client
so
that
we
can
see
what
happens
when
you
try
to
switch
rights
from
an
unsharded
cluster
to
a
sharded
one.
So
I'm
going
to
start
this
client,
which
will
just
write
some
rows
into
the
main
key
space,
and
once
that
starts,
we
will
switch
rights.
A
A
A
A
A
A
A
A
We
have
a
number
of
talks
at
the
conference.
There
were
two
yesterday
and
if
you
miss
them,
you
might
want
to
catch
up
with
the
recordings,
and
there
are
two
more
talks
coming
up.
One
is
deep
dive
into
witness
and
the
other
one
is
a
multi-cloud
with
a
stock,
and
these
are
happening
in
about
an
hour
or
so.