►
From YouTube: Geo-partitioning with Vitess - Deepthi Sigireddi, PlanetScale & Jitendra Vaidya, PlanetScale
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Geo-partitioning with Vitess - Deepthi Sigireddi, PlanetScale & Jitendra Vaidya, PlanetScale
Geo-partitioning is an emerging requirement for multinational corporations. Laws such as the GDPR in Europe stipulate where a user’s data needs to be physically located, as well as when it can be transferred out of the EU. We expect additional legal jurisdictions to pass similar laws in the coming years, and that database management systems will require native functionality to assist in compliance. In the case of Vitess, one of its distinguishing features is support for flexible sharding schemes. This can easily be extended to support a custom sharding scheme that respects geo-partitioning requirements. In this deep dive we will first explore how and then demonstrate a database cluster built using the custom sharding scheme that solves data residency at the database layer obviating the need for any change at the application layer.
https://sched.co/UagI
A
A
A
B
A
A
So
here
is
the
problem.
Right,
like
you,
guys
know
that
different
jurisdiction
are
passing
laws
that
say
that
the
data
for
users
which
belong
to
that
particular
jurisdiction
should
not
leave
the
jurisdiction.
We
saw
this
with
in
Europe
with
GDP.
Our
California
has
now
passed.
A
similar
legislation
will
which
will
go
into
effect
fairly
soon,
and
so
many
different
people
are
thinking
about
how
to
support
this
and
most
of
the
approach
involved.
Basically,
RER
collecting
your
applications
where
you
introduce
the
notion
of
which
jurisdiction
a
particular
users.
Data
belongs
to
at
the
application
layer.
A
What
we
are
proposing
is
that
you
and
you
sort
of
encapsulate
that
logic
in
Vitesse,
so
that
your
application
doesn't
need
to
be
aware
of
it
and
that's
what
we
are
going
to
show
you.
Similarly,
your
you
have
existing
monolithic
databases
that
are
sitting
somewhere.
Maybe
some
of
them
are
in
the
jurisdiction
that
you
want.
Some
of
them
are
not
and
migrating
those
databases
now
and
also
maybe
splitting
the
data,
because
you
might
have
users
which
span
multiple,
multiple
jurisdictions.
So
how
do
you
migrating
those
databases
into
jurisdiction?
Aware
databases
becomes
an
operational
nightmare.
A
So
how
do
we
solve
that?
So
those
are
the
problems
that
we
are
going
to
talk
about
today,
you
using
Vitas
so
the
way
we
have
structured.
The
talk
is
that
I'm
going
to
first
start
out
by
talking
about
a
little
bit
about
Vitas
different
concepts
in
Vitesse,
and
once
you
have
enough
of
an
understanding
of
the
concepts
that
we
will
use
to
solve
these
problems,
we
will
talk
to
you
about
the
solution
and
then
Deepti
is
going
to
do
a
demo
that
actually
shows
you
four
clusters
running
in
four
different
kubernetes.
A
Well,
a
key
space
that
runs
across
for
kubernetes
clusters
in
four
different
parts
of
the
world
in
which
we
write
the
data
and
you
can
see
it
doing
the
right
thing.
So
that's
the
demo
that
dipti
is
going
to
show
you
that's
how
we
have
structured
the
talk.
So
this
is
the
Vitesse
architecture
diagram
that
you
all
know
and
love.
Most
of
you,
VT
gate
is
the
stateless
proxy
supports.
My
sequel,
binary
protocol
has
a
full
SQL.
A
Parser
gives
your
application
the
view
that
it's
talking
to
a
single
humungous
database
when
behind
it
are
multiple
shards.
So
what
a
shard
is
what
a
key
space
is.
All
of
that
we
will
be
going
all
those
concepts
we'll
be
talking
about
in
further
detail
later,
because
all
those
concepts
you
need
to
understand
to
understand
how
to
architect
these
clusters,
but
each
shard
is
basically
a
mass
cluster
with
one
master,
multiple
replicas
replicating
using
my
sickle
binary
protocol
each
my
security
gets
a
minder
process.
A
That's
called
beauty
tablet
which
protects
at
my
security
and
also
does
many
other.
You
know
it
helps
you
during
the
parenting.
It
helps
you
during
backups
a
lot
of
those
things.
Topology
server
stores,
two
different
things:
one
is
the
the
topology
in
terms
of
the
IP
address,
host
IP,
address
port
numbers
of
the
various
processes,
and
the
second
is
description
of
how
each
table
in
a
Charlotte
key
space
is
charted
right.
So
those
two
that
two
two
types
of
information
in
the
topology
server
VT
CTL-
is
the
control
plane.
A
So
once
again,
this
is
the
architecture
of
the
Vitesse.
Keep
this
diagram
in
your
mind
as
I
talk
about
the
various
concepts,
so
the
first
concept
I
am
going
to
talk
about
itself,
so
this
diagram
does
not
show
you
a
cell,
but
each
of
these
shards,
which
has
one
master
and
multiple
replicas.
They
need
to
be
residents
somewhere
right.
A
They
need
to
be
resident
in
some
bit
of
servers
and
network
infrastructure,
be
kubernetes,
cluster
region
and
availability
zone.
So
basically
a
cell,
for
wit,
s,
is
a
failure,
domain,
isolated
from
failures
and
other
cells,
which
is
a
group
of
servers
and
network
infrastructure
right.
So
an
example:
if
you're
running
on
prem
could
be
a
full
data
center
or
even
a
rack
in
a
data
center
right
or
it
could
be
an
availability
zone
in
the
region,
it
could
be
a
region
itself
or
it
could
be
a
kubernetes
cluster.
A
So
anything
which
is
so
what
I
want
to
point
out
is
that
a
cell
basically
has
physical
properties
to
it.
So
it
is
resident
in
some
physical
location,
which
means
that
its
resident
in
a
jurisdiction-
and
so
we
are
going
to
use
this
property
to
figure
out
how
to
build
these
clusters.
Now,
let's
talk
a
little
bit
about
a
key
space.
So
in
a
Charlotte
system,
a
data
like
Vitesse,
we
call
a
database
key
space.
So
if
your
key
space
has
a
single,
then
a
key
space
is
no
different
from
my
sequel
database.
A
A
So
in
either
case
the
key
space
appears
is
a
single
database
from
the
standpoint
of
the
application.
Now,
let's
talk
about
what
a
key
space
ID
is.
This
is
a
little
bit
of
an
interesting
concept.
Those
of
you
who
have
worked
with
key
value
stores
probably
know
that
a
key
space
ID
is
what
a
key
value
store
typically
uses
for
sharding
right,
but
in
Vitesse
we
never
actually
store
a
key
space
ID
for
every
row.
So
a
key
space
is
split
up
into
key
space.
Id
ranges
which
are
shards.
A
Each
row
is
assigned
a
key
space
ID,
which
is
sort
of
the
street
address
of
a
row
so
using
the
key
space
ID
bit
as
figures
out,
which
shard
a
particular
row,
should
go
into
right.
It's
internal
to
witness.
The
application
doesn't
need
to
know
about
it
and
it's
not
stored
but
computed,
and
that
gives
us
a
lot
of
flexibility
and
we'll
talk
about
this
a
little
further.
Now,
let's
talk
about
what
of
indexes
of
index
is
how
you
tell
so.
We
have
extended
the
relational
semantics
and
the
way
relational
databases
have
schemas
Charlotte
databases.
A
A
Is
the
column
that
you
want
to
use
for
showering
that
particular
table,
and
the
second
is
the
sharding
function
that
you
want
to
use
since
we
allow
you
to
arbitrarily
choose
any
column
to
as
the
sharding
column,
for
a
table
depending
on
the
type
of
that
column,
you
might
want
to
use
different
function.
So
if
it's
a
numeric
column,
you
would
typically
want
to
use
hash.
If
it's
a
you
know,
you
are
care
text
or
something
like
that.
You
will
probably
want
to
use,
lose
md5
hash,
or
something
like
that.
A
So
the
example
that
I'm
giving
you
here
is
that
let's
say
that
there
is
a
table
called
customer
and
you
want
to
shout
it
using
ID,
which
happens
to
be
a
numeric
column,
and
you
want
to
the
shouting
function
that
you
want
to
use
is
hash.
So
for
a
row
whose
ID
is
1,
2
3,
the
key
space
idea
would
be
hash
of
1
2,
3,
pretty
straightforward
right
again:
computed
not
stored.
A
What's
a
char,
so
a
char
basically
has
a
begin
and
an
end
value
for
key
space
IDs.
So
if
we're
test
wants
to
figure
out
whether
a
particular
row
belongs
in
a
particular
char,
all
that
it
has
to
do
is
to
evaluate
that
condition.
Begin
less
than
keyspace
ID
less
than
equal
to
n,
and
that
that
gives
you
the
answer.
If
this
holds
true,
then
the
row
for
which
it
has
computed
that
key
space
ID
belongs
in
that
char.
A
Otherwise
it
doesn't
right
and
a
key
space
is
basically
a
connect
collection
of
shards
shards
where
the
beginning
of
the
loop,
where
the
key
space
idea
ranges
and
the
whole
key
space
right.
That's
why
it's
called
a
key
space.
Each
char
has
one
master,
multiple
replicas,
so
each
shard
is
a
traditional,
my
sequel,
cluster
right
and
these
masters
and
replicas
can
be
located
in
one
or
multiple
cells
right.
A
So,
let's
have
you
keep
all
this
in
your
head
and
then
now,
let's
talk
a
little
bit
about
shouting
functions,
so
I
told
you
that,
depending
on
the
type
of
the
column,
you
can
choose
different
shouting
functions
right,
and
so
these
seven
functions
are
predefined.
Shouting
functions
in
Vitesse
and
I
talked
about
hash
and
I
talked
about
the
Unicode
loose
md5
which
could
you
would
want
to
use,
for
you
know,
text,
text,
columns
and
so
on,
but
you
can
also
add
your
custom
shouting
function
and
that's
that's
the
coolest
thing
about
witness.
A
As
long
as
you
get
your
shouting
function
right
so
you're
you,
you
can
create
your
shouting
functions
and
then
you
can
expand
refer
to
them
in
the
JSON
that
you
use
to
tell
Vitesse
about
how
you
are
shouting
defining
your
v--
indexes.
So
that's
those
are
the
shouting
functions
so
now
that
we
have
talked
about
this
concept
so
before
I
start
talking
about
the
solution,
how
we
solve
this
problem
of
a
jurisdictional
databases.
Let
me
a
spawn
here
and
ask
you
if
you
have
any
questions
about
what
we
have
talked
so
far.
A
A
A
Where
a
particular
user,
or
where
a
particular
particular
piece
of
data
lives
such
as
a
country,
a
zip
code,
a
state
or
an
area
code,
that
would
be
the
second
level.
So
what
we
mean
by
that
here
is
that
examples
of
jurisdictions
are
EU,
NAFTA
or
California
right.
So
let's
say
that
if
we
were
trying
to
solve
the
problem
for
California's
data
locality
laws,
the
two
jurisdictions
that
I
would
have
if
most
of
my
users
are
in
in
in
the
US,
would
be
California
and
non
California.
A
Those
would
be
my
two
jurisdictions
and
then
probably
the
geographical
location,
which
is
represented
by
a
data
column,
can
either
be
an
area
code
or
could
be
a
zip
code
it.
Basically,
it
allows
me
to
locate
that
particular
piece
of
data
into
a
particular
jurisdiction
in
our
case
California
or
non
California.
In
the
example
that
we
have
shown
here,
we
have
four
jurisdictions:
North
America,
South,
East,
Asia,
East,
Asia,
South,
Asia,
East,
Asia
and
Europe.
These
are
sort
of
you
know.
A
They
don't
correspond
to
any
geopolitical
realities,
but
that's
what
we
have
used,
but
so
two
levels
level
of
jurisdictions
that
we
test.
Doesn't
know
anything
about
some
column
in
some
table
that
you
want
to
shard,
which
can
help
us
locate.
That
particular
piece
of
data
in
our
jurisdiction
is
the
second
level,
and
what
we
do
is
we
locate
different
shards
in
a
Cell
bounded
by
jurisdiction.
That's
that's
how
we
solve
this
problem.
A
B
Before
we
actually
start
showing
you
the
demo,
we
want
to
what
the
goals
of
this
demo
are
and
so
that
we
can
demonstrate
the
jurisdiction
of
a
database
that
we
are
trying
to
show
here.
So
for
this
demo,
we've
created
a
witters
cluster
that
stores
data
for
a
given
country
in
its
jurisdiction.
The
client
application
doesn't
need
to
know
that
data
is
stored
in
multiple
locations
or
where
exactly
the
data
is
stored.
So
it's
completely
transparent
to
the
client
for
the
demo
itself.
B
The
way
we've
designed
the
cluster
is
that
we
have
created
a
witness
cell
for
each
of
these
jurisdictions
and
we've
named
them
according
to
the
region
in
Google
Cloud,
where
that
cell
lives.
So
there's
us
central
one,
a
Europe
s,
three
a
is
one
a
in
Asia
South
to
a
because
we
are
going
to
show
the
demo
on
GCP
or
rather
gke.
B
B
So
this
is
a
multi-column
Windex
that
uses
more
than
one
column,
so
it
uses
what
would
have
been
your
sharding
key
if
you
didn't
have
the
jurisdictional
restrictions
and
an
additional
column
that
represents
the
geographic
location.
That
is
restricting
you
in
terms
of
where
you
can
locate
the
data
and
you
take
the
these
two
columns.
In
this
case
they
are
ID
and
country,
and
you
map
that
tuple
to
a
key
space
ID,
which
you
store
in
a
lookup
table.
People
here
that
are
familiar
with
with
us,
already
know
what
lookup
indexes
are.
B
They
are
somewhat
different
from
the
wind
X's
you
use
for
your
sharding,
and
this
is
a
combination
of
a
lookup
index
and
a
sharding
function
that
we
are
using
for
doing
the
jurisdiction,
aware
clusters,
the
region
Windex,
looks
up
the
region
byte
for
a
country
using
a
map
and
the
source
code
for
this
is
available,
and
you
can
follow
the
link
to
look
at
that.
I
won't
go
into
that
now.
B
How
do
we
map
the
key
space
to
jurisdictions?
We
have
split
the
one
key
space.
We
have
user
DB
into
four
key
ranges.
Key
ranges
are
basically
subsets
of
the
entire
key
space
which,
when
combined
together,
span
the
entire
key
space,
so
they
are
disjoint,
but
the
Union
gives
you
the
full
key
space,
so
the
first
shard
40
corresponds
to
Americas.
Then
we
have
40
to
80.
B
These
are
hex
numbers
by
the
way
corresponds
to
Europe
80
C
0
corresponds
to
East
Asia
and
C
0
C
0,
to
whatever
is
South
Asia
and
within
each
of
these
key
ranges,
we
have
assigned
specific
values
for
the
byte
that
we
will
use
to
locate
the
data
based
on
the
country.
So
us
is
one
Canada,
as
France
is
4.
0
Germany
is
4,
1
and
so
on.
B
B
We
have
some
sample
data
that
we
will
insert
into
the
scroll
into
the
database,
that's
resident
in
the
cluster
that
we've
created
in
the
data.
We
have
some
personally
identifying
information
named
national
ID.
These
are
things
that
are
usually
subject
to
loss
that
govern
data,
residency
and
cross-border
data
transfers,
and
we
have
the
column
that
tells
us
where
this
should
live.
The
location
of
the
data.
B
B
C
B
We
look
at
the
schema
for
these
key
spaces
right
now.
There
is
no
schema.
There
is
no
data,
there's
nothing,
it's
basically
empty,
so
we
will
start
with
this
empty
database
or
key
space
and
then
create
a
schema.
Create
a
V
schema.
V
schema
is
what
tells
with
us
how
you
would
like
to
shut
this
key
space
and
then
insert
some
data
into
the
database
and
see
where
it
goes.
You
might
notice
that
I
hope
that's
more
visible.
B
Here
we
have
the
schema
and
we
schema
that
we
will
be
deploying
into
this
cluster.
In
the
user
DB
key
space.
We
have
just
one
table
customer.
There
is
an
ID
which
we
will
Auto
Inc.
We
will
use
an
auto
increment
for
populating
the
ID
and
the
name
of
the
customer
or
the
national
ID,
and
the
country
that
we've
already
decided
to
use
here
is
the
V
schema
in
the
V
schema.
We
are
specifying
that
this
is
a
sharded
key
space
and
that
there
are
certain
wind
X's.
B
We
have
just
one
Windex
called
region
v
DX
and
the
type
of
this
Windex
is
region
Windex
and
this
region
Windex
that
we
have
written
the
custom.
Windex
takes
certain
parameters.
There
is
a
map
that
you
use
to
map
unique
values
for
each
of
the
location,
identify
us
in
our
case
it's
country,
so
we
have
called
it
country's
dot
Jason.
B
You
also
specify
how
many
bytes
of
the
key
space
ID
you
would
like
to
use
for
the
region
when
we
use
country
one
bite
is
sufficient.
But
if
you
were
to
do
sip
code
or
area
code,
as
Jatin
mentioned
as
possibilities,
then
you
might
need
to
use
more
than
one
byte
and
this
Windex
was
written
to
allow
you
to
use
one
or
two
bytes.
That's
an
implementation
time
decision
and
the
Windex.
B
You
use
this
an
auxilary
table
to
store
the
mapping
of
the
ID
and
country
to
the
key
space
ID,
and
this
Windex
is
owned
by
the
customer
table.
Now,
let's
look
at
the
customer
tables.
B
schema.
The
customer
table
has
a
column
Windex
just
one
and
the
columns
that
the
Windex
is
based
on
our
ID
and
country,
and
the
name
of
the
Windex
is
region
vdx
that
we
defined
up
here.
B
B
The
lookup
key
space
is
not
sharded
and
technically
you
don't
even
have
to
specify
a
V
schema
when
it's
an
uncharted
key
space,
but
just
for
completeness,
we
have
included
what
the
V
schema
should
look
like
for
the
Uncharted
key
space.
That
includes
the
sequence
now
that
we
know
what
our
schema
and
V
schema
look
like.
Let's
deploy
them.
B
Then
we
will
apply
the
schema
to
lookup
key
space
and
apply
V
schema
to
the
lookup
key
space,
and
once
that's
all
done,
we
can
go.
Look
at
what
that
looks
like.
So,
let's
refresh
this
view
so
now
the
lookup
key
space
has
two
tables
the
sequence
table
and
the
mapping
table
and
the
mapping
table
has
no
data.
There's
zero
rows
and
there
are
no
wind
X's
in
this
key
space
because
it's
uncharted.
B
B
A
B
B
B
40
to
80
has
4
rows,
80
to
C,
0
has
4
rows
and
so
on.
Each
of
the
shards
has
now
now
has
4
rows
of
data
in
the
table.
One
thing
to
note
about
this
control
panel
UI,
is
that
the
number
of
rows
that
we
show
comes
from
the
information
schema,
so
it
is
not
always
guaranteed
to
be
exact
depending
on
how
fast
you
are
writing
to
the
database.
It's
only
an
approximate
number,
but
because
here
we
are
writing
such
a
small
number
of
rows
and
we
don't
have
continuous
traffic.
B
B
Let's
examine
the
data
one
shot
at
a
time,
so
witness
allows
you
to
target
your
queries
to
specific
key
spaces
and
also
to
specific
shards
within
a
key
space.
So
I
could
say,
use
user,
DB
and
I
would
only
see
the
data
which
is
in
this
key
space.
I
would
not
be
able
to
see
the
data.
That's
in
the
lookup
key
space
and
I
can
also
say
use
user
DB,
a
specific
shard,
and
this
will
show
me
the
data
that
resides
in
that
specific
shard.
B
B
B
B
C
B
B
B
The
question
is:
can
you
do
joints
between
the
regions
in
general?
In
witters?
You
can
do
cross
shard
joints
when
you,
when
you
are
trying
to
achieve
data
locality,
then
all
data
that
has
to
reside
in
that
region
better
stay
in
that
region,
in
the
sense
that
you
shouldn't
be
trying
to
join
that
with
another
region.
But
you
might
have
some
some
table
that
is
relevant
to
all
regions
like
maybe
the
pricing
table
and
yeah
you
can
do
a
join
across
charts,
which
means
you
can
do
joins
or
cross
regions
right.
A
So,
if
I
may
do
a
little
bit
of
a
clarification,
for
example,
let's
say
that
you
had
an
order
stable
right.
The
orders
table
also,
you
would
shard
it
using
the
ID
of
if
you
started
using
the
user
ID,
because
in
orders
also
will
have
a
user
ID
column
in
it.
Then
we
will
also
correctly
look
at
the
data
in
the
correct
shard,
and
so
the
users
row
and
users
orders
will
all
be
in
the
same
shot.
So
the
joints
will
end
up
happening
in
the
same
shot.
B
If
we
had
more
time,
we
would
have
liked
to
show
a
lot
more
on
what
you
can
do
with
jurisdiction.
Aware
clusters,
you
could
start
with
a
monolithic
database,
which
already
has
all
of
your
data
and
rashard
it
and
push
it
out
into
the
various
regions.
You
might
want
to
move
a
country
from
one
jurisdiction
or
cell
to
another
brexit
or
you
might
want
to
break
a
region
into
smaller
shots.
Just
because
you
have
too
much
data
and
say
the
East
Asia
region
or.
A
B
Before
we
conclude,
there
are
a
few
people
we
want
to
thank
for
making
this
demo
possible.
Dan
who's.
Our
minister
of
engineering
at
planet-scale,
was
the
one
who
first
suggested
the
idea
for
the
stock,
and
that
was
the
day
after
we
were
done
with
cube
con
Europe
in
Barcelona
six
months
ago,
and
six
months
later,
here
we
are
with
the
demo.
B
B
B
B
A
So
there
is
a
shards
to
watch
command
line,
parameter
to
VT
gate,
and
you
can
specify
that
a
particular
witta
gate
only
it
looks
at
a
particular
shard,
and
so
some
and
each
of
them
can
have
a
different
authentication,
username
and
password
configured,
and
so
in
this
somebody
who
has
those
you
know
those
credentials
can
connect
to
that
particular
vita
gate
which
can
look
at
the
particular
shard.
That
could
be
one
way
of
doing
it.
If
that's
what
you
are
getting
to
by
your
question,
if.
B
All
of
your
data
was
sharded,
then
you
would
be
able
to
just
make
witty
gate
watch
that
single
cell,
so
that
anyone
who's
connecting
in
that
region
is
only
able
to
access
data
that
belongs
to
that
region.
But
typically
most
people
have
some
tables
sharded
and
some
tables
uncharted,
which
complicates
it.
There
was
another
question
right.
B
B
B
B
B
B
B
Yes,
so
the
question
is,
we
haven't
talked
about
performance,
what,
if
the
applications
are
also
located
in
all
of
the
regions?
What
effect
does
it
have
on
performance
in
the
even
in
this
demo
that
we
deployed,
we
have
Rita
gate,
the
stateless
proxy
deployed
into
every
region,
so
the
application
can
be
configured
to
talk
to
the
VT
gate
or
VT
gates
behind
a
load
balancer
in
the
same
region,
and
that
should
minimize
the
latency.
B
So
the
question
is:
is
it
going
to
happen
locally?
Is
all
the
data
going
to
be
written
only
to
the
region,
that's
relevant?
If
all
of
the
data
is
in
one
region,
then
that
minimizes
your
latency,
but
if
you
do
have
some
data
that
you
have
to
access
in
a
different
region
for
whatever
reason,
keeping
in
mind
the
cross-border
data
transfer
requirements,
then
that
will
add
latency.
B
B
B
A
So
order
by
is
a
simple
query,
so
what
we
do
is
we
send
the
order
by
to
each
my
sequel
and
do
a
merge
sort
at
the
Vedic
eight
level
and
show,
but
there
may
be
some
complex
queries
where
you
are
doing.
Group
buys
and
order
buys
which
need
all
rows
in
VT
gate
to
satisfy
that
query
correctly,
and
we
just
don't
support
it.
There
is
a
list
of
queries
which
we
don't
suffer
bit
exactly
because
of
that
reason.
B
A
Well,
no,
so
the
thing
is
that
if
you
have
stored
procedures
which
only
work
on
data
that
is
in
a
single
shard,
they
will
continue
to
work.
Waitis
itself
doesn't
support
it,
but
you
can
create
them
by
connecting
to
the
my
security
by
itself
and
as
long
as
their
it
works
on
the
in
the
context
of
a
single
shard.
They
continue
to
work,
but
we
don't
have
support
for
stored
procedures
that
work
correctly
in
data
across
shards.