►
From YouTube: Webinar: Vitess: Sharded MySQL on Kubernetes
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Vitess has continued to evolve into a massively scalable sharded solution for the cloud. It's is now used for storing core business data for companies like Slack, Square, JD.com, and many others. This webinar will cover the high-level features of Vitess with a focus on what makes it cloud-native. We'll conclude with a demo of the powerful materialized views feature that most sharded systems have yet to solve.
A
All
right
so
hello
and
welcome
again
everybody.
This
is
the
the
test
started,
my
sequel
on
kubernetes
seminar
and
my
name
is
Randi
Abernathy
I
am
a
CNC
F
ambassador
and
I'm,
a
managing
partner
at
our
XM
cloud
native,
consulting
and
training
shot
and
we're
basically
here
to
hear
sue
sue.
Morons
discussion
about
the
test.
Sugu
was
one
of
the
co-creators
of
the
tests
at
youtube
and
now
is
the
CTO
of
planet-scale
and
so
sue
cool.
If
you
would
like,
please
take
it
away
thanks.
B
It
was
actually
born
out
of
the
need
that
we
had
at
YouTube,
where
we
were
managing
many
mice
equal
instances,
and
it
was
beginning
to
go
out
of
control.
There
was
just
too
many
outages
and
you
were
up
against
the
wall,
so
Mike
my
co-creator
and
I
decided
to
think
about
what
can
we
do
to
leap
ahead
of
all
the
problems
that
we
had?
We
were
currently
facing,
and
essentially
that's
how
we
test
was
born
a
quick
background
on
what
we
test
is
for
those
of
you
who
may
not
be
familiar
with.
B
B
What
that
means
is
this
that
if
you
install
witness
on
your
my
sequel
instance,
then
you
can
shard
it
underneath
and
as
far
as
the
application
is
concerned,
you
get
the
impression
that
you
are
still
talking
to
a
you
know
if
database
and
that's
a
huge
win
for
a
lot
of
people,
because
without
something
like
the
tests,
you
have
to
make
pretty
big
changes
to
your
app.
If
you,
if
you
want
the
shoddier
database,
so
that
is
one
big
problem
that
with
tassels
yeah,
because
of
the
fact
that
we
were
running
it
at
YouTube.
B
It
is
naturally
massively
scalable.
So
any
solution
that
we
came
up
with
had
to
scale
into
millions
of
QPS
and
tens
of
thousands
of
nodes,
and
also
it
had
to
be
highly
available
available,
which
means
that
down
time
beyond
a
few
seconds
is
something
that
was
something
that
someone
something
like
YouTube
would
not
tolerate.
So
we
toss
comfortably
gives
you
five
nines
of
availability
in
most
of
the
production.
B
So
these
are
some
of
the
stats
about
B
test.
Like
I
said
it
was
started
in
2010
and
has
been
gaining
popularity
so
initially,
even
when
we
decided
to
start
with
tests,
we
open
sourced
it,
but
it
wasn't
really.
The
impact
wasn't
really
to
have
other
people
adopted
because,
mostly
because
we
wanted
to
expose
the
work
that
we
had
done
and
also
not
have
to
reinvent
that
wheel.
If
we
went
somewhere
else
right,
and
so
it
was
mostly,
we
were
submitting
code
into
the
open
source,
but
just
it
was
mostly
just
for
YouTube's
own
sake.
B
We
used
to
just
import
that
code
back
into
YouTube
and
then
launch
it
inside,
but
around
2000,
fifteen
or
sixteen
Flipkart
found
us,
and
then
they
said
hey.
It
looks
like
this
is
something
that
we
can
also
use,
and
so
then
they
start
they
worked
on
adopting
it
and
over
time
more
and
more
companies
came
and
today,
actually
the
the
biggest
pride
of
the
test
is
who
are
the
adopters
of
with
us?
We
have
a
pretty
impressive
list
of
technology.
Leaders
that
are
adopting
a
witness
like
here
is
some
example.
B
Slack
is
actually
a
major
contributor
to
with
us
and
also
a
major
adopter.
Their
intent
is
to
actually
fully
migrate
to
with
us
and
I.
Think
their
goal
is
to
do
it
by
end
of
this
year.
At
this
point
they
are
30%
migrated.
They
actually
talked
about
some
of
their
stats
in
our
last
meet
up
just
a
few
days
back.
They
say
that
they
peaked
at
1.2
million
QPS
and
they
even
gave
some
number
of
shards
I
think
it
was
in
the
high
hundreds
or
maybe
a
thousand.
B
It's
not
shards
thousand
instances,
nodes
of
test
nodes
square
cache
actually
is
the
classic
case
of
with
us
use
where
they
had
a
single
instance,
my
sequel
that
was
running
on
the
biggest
Hardware
they
could
buy,
that
money
could
buy
and
that
was
running
out
of
capacity
and
what
they
did
was
they
launched
with
us.
On
top
of
that
huge
database
and
then
charted
it
underneath
I
believe
they
they're
now
at
the
last
number
was
that
they
were
at
32
shards
and
we're
looking
at
going
to
64.
B
So
the
one
of
their
statement
is
that
sharding
is
addictive,
because
once
you
shard
things
run
battle,
and
then
you
say:
oh,
let's
start
some
more
and
let's
start
some
more,
so
it
just
keeps
going
and
you-
and
it's
mostly
a
lot
of
gain
with
very
little
a
little
loss
and
Pinterest
actually
recently
migrated
to
with
us
one
of
the
cool
quotes
is
likely.
Sharding
is
eventless
almost
boring,
so
that's
one
thing
that
we
test
makes
really
really
easy
for
you
to
do
so.
B
Here's
another
thing
so
Kashyap
actually
had,
which
is
from
square,
grew
so
fast
that
Jack
Dorsey
himself
quit
about
it.
So
now
the
top
two
apps
and
the
top
charts
are
using
with
test
and
we
are
going
after
Burger
King
next
cool.
So
here
is
a
list
of
some
of
the
bigger
doctors
that
we
test.
This
slide
actually
means
to
be
updated.
There
are
a
few
more
users
that
have
that
are
beginning
to
come
on
board,
but
we
need
to
get
them
to
list
their
logos
logos
in.
B
Yet
these
are
the
people
that
have
agreed
to
have
their
logos
listed.
The
most
important
takeaway
from
this
is
that
with
test
runs
in
a
large
number
of
environments,
because
each
of
these
companies
runs
has
a
different
production
setup.
Some
are
on
bare
metal.
Some
are
on
cloud
like
running
ec2
instances.
Some
are
running
kubernetes.
B
Some
are
running
other
cloud
orchestration
software.
There
is
at
least
one
Vitesse
instance,
I
believe
that
runs
on
ASOS
and
there
is
another
user.
That's
looking
at
running
it
on
Nomad
so,
and
there
is
a
reason
for
this
is
because
which
I
will
go
into
which
which
is
somehow
related
to
making
with
test
cloud
native
also.
B
So
why
is
with
that
cloud
native?
Why
it
does
with
us
claim
to
be
so
compatible
with
kubernetes
is
what
we
are
going
to
talk
about.
So
this
goes
into
the
history
of
with
us.
So
when
we
first
developed
with
us,
we
were
it
was
actually
developed
only
for
bare
metal,
but
in
2013
what
happened-
and
we
were
running
with
this
on
bare
metal
in
YouTube's
own
data
centers.
B
We
were
not
part
of
Google
data
centers
in
2013,
Snowden
happened
and
Google,
pretty
much
mandated
that
we
are
not
allowed
to
have
data
sensitive
data
outside
of
Google
data,
centers
itself
that
we
had
to
migrate
everything
inside
Google's
cloud
and
that
faced
a
pretty
big
challenge,
because
we
test
was
open
source
and
we
were
running
it
on
this
bare
metal
and
I.
Don't
know
how
many
of
you
have
worked
at
Google,
but
if
you
develop
software
that
runs
inside
Google's
ecosystem,
it
is
pretty
much
almost
impossible
to
open
source
it.
B
The
reason
is
because
Google
has
so
many
custom
api's
right
there
is
chubbie,
there
is
stubby
there's
all
these
logging
API
is
monitoring
api's
and
they
are
all
internal
to
Google.
So
if
you
write,
if
you
wrote
software
that
ran
inside
the
Google's
Borg
cloud,
that
is
pretty
much
not
open-source,
but
when
we
decided
to
port
with
tests
into
Google's
borwein
face
to
challenge,
we
said:
okay,
do
we
just
close
it?
B
And
then
we
like
we
test
to
use
Google's
internal
api's,
or
do
we
try
to
keep
it
open
source
and
somehow
make
it
work
inside
Google's
cloud?
So
we
went
for
the
latter
decision
and
it
was
really
really
hard,
because
what
that
meant
was
actually
building
abstraction
layers
for
every
API
of
Google
that
we
were
using
like
throttling
and
whatever
else
that
that
was
needed
right
like
calling
into
Balkman
calling
it
to
chubbie
calling
into
stubby.
B
So
we
actually
built
abstraction
layers
for
each
of
those
things,
and
that
is
how
we
finally
managed
to
get
it
working
inside
Borg,
so
one
cool
property
of
with
us
that
even
I
believe
not
even
a
storage
system
inside
Google
has
his
Vitesse
ran
as
a
stage
less
app
in
google's
borg,
which
means
that
we,
just
as
far
as
Google's
cloud
was
concerned
with
those
was
just
an
app.
It
was
not
a
special
storage
system
with
any
privileges
about
what
it
could
do
with
storage.
B
B
Kubernetes
1.0
came
out
in
july,
but
we
actually
announced
that
Vitesse
was
ready
for
koukin,
it
is
before
kubernetes
1.0
came
out
and
we
went
out
and
told
people
that
you
can
use
with
us
on.
Kubernetes
and
people
started
using
it
actually
stitch
labs
was
actually
the
first
company
to
adopt
and
they
were
pretty
running.
They
are
a
small
company,
but
they
are
running.
B
Then
came
HubSpot.
Well,
the
big
word
that
they
did
was
they
actually
did
a
lot
of
compatibility
work
to
make
a
large
number
of
queries
work
then
JD
dot-com
came
and
they
actually
scaled
it
massively
also.
So
these
these
companies
have
been
running
with
us
on
kubernetes
for
many
years
now.
So
it
is
not
a
theory.
B
B
They
started
on
Azure
because,
as
you
gave
them
huge
credits
and
eventually
they
ran
out
of
those
credits
and
within
one
hour
they
migrated
from
Azure
into
Google,
Cloud,
GCP
and
I,
don't
like
their
learning
gke
or
their
own
kubernetes,
most
likely
gke.
But
that
is
how
easy
it
is
for
you
to
migrate
from
one
cloud
to
another
if
you
are
completely
on
kubernetes
end-to-end.
So
that's
that's
one.
B
B
But
then,
after
all
this
here,
Kelsey
keeps
saying:
don't
move
your
storage
to
kubernetes,
it
is
problematic,
run
your
stateless
upon
kubernetes,
but
storage
I
would
not
recommend
right,
but
there
is
a
reason
why
he
says
that
he
is
because
of
this.
Essentially,
what
he
is
saying
is
if
you
just
took
a
storage
system
and
just
moved
it
to
kubernetes
you're,
going
to
regret
it
because
there
are,
there
are
things
missing
there.
There
are
gaps
between
kubernetes
and
storage
layer
that
need
to
be
filled
and
existing
storage
systems
just
don't
run,
as
is
on
kubernetes.
B
So
that's,
essentially
what
you
are
trying
to
say
and
to
explain
why
we
test,
which
is
the
gap
I'm
going
to
show
you
the
Vita's
architecture
here.
The
way
we
test
works
is
even
though
it
can
scale
really
really
massively.
The
witness
architecture
is
fairly
simple:
it's
just
a
two
layer
architecture.
You
can
see
the
app
server
here
it
connects
to
these
VT
gates.
These
mitigates
are
stateless,
app
servers,
so
it
can
connect
to
any
other
through
a
load
balancer
and
as
far
as
the
app
server
is
concerned,
a
mitigate
represents
your
entire
cluster.
B
It's
as
if
it's
connected
to
a
single
my
sequel
instance,
but
under
the
under
the
covers
what
we
decay
does
is
when
you
send
a
query
to
it.
It
actually
has
a
metadata
called
the
V
schema,
which
is
kind
of
a
schema,
but
for
a
sharded
system,
and
it
uses
that
schema
to
figure
out
where
that
query
should
go
and
then
once
it
figures
it
out,
it
sends
it
to
a
bt
tablet
and
each
VT
tablet
runs
its
own.
B
My
sequel
instance,
and
what
this
VD
table
does
is
actually
completely
manage
that
my
sequel
instance,
and
there,
and
also
it
has
other
benefits
like
connection
pooling.
It
can
also
do
housekeeping
work
like
backups
and
restores
and
those
kinds
of
things
you
can
see
here
that
there
are
many
types
of
meaty
tablets.
There
is
a
master
meaty
tablet
which
accepts
writes,
and
there
are
some
replicas
VD
tablets
that
accept
read
traffic,
etc.
B
So
this
is
one
case,
one
architecture
which
you
cannot
directly
get
if
you
move
your
my
sequel
to
kubernetes.
So
let's
say
you
want
to
move.
You
want
to
deploy
my
sequel
on
kubernetes,
so
gubin.
It
is
the
the
best.
The
best
part
type
to
use
is
a
stateful
set
right.
The
stateful
set
a
workload
gives
you
the
ability
to
have
a
bunch
of
instances,
but
what
the
stateful
set
does
not
allow
you
to
do
is
designate
one
of
those
instances
as
a
master.
B
As
far
as
kubernetes
is
concerned,
they
are
all
equal.
So
if
you
want
to
designate
one
part
as
a
master,
what
you
have
to
do
is
actually
pull
that
out
into
a
separate
part
type,
and
if
you
did
that,
then
that
part
becomes
a
master
and
then
the
rest
of
the
parts
become
replicas.
So
then
you
can
say:
okay,
it's
a
direct
traffic
to
the
master
part
direct
traffic
to
the
replicas.
But
then
what
happens?
B
If
a
mass
will
goes
down
right,
so
master
goes
down,
then
you
cannot
fail
over
to
a
replica
and
say
now.
This
part
is
designated
at
mass
as
a
master,
so
kubernetes
doesn't
allow
you
to
change
a
part
type
on
the
fly,
which
means
that
you
have
to
wait
for
the
master
to
be
restarted
so
kubernetes
to
restart
that
part
as
a
master
and
then
once
it
comes
back
up,
then
you
can
send
traffic
to
it.
B
It
can
actually
track
where
the
master
is
and
if
a
master
goes
down,
we
can
repair
it
and
designate
existing
replicas
as
a
master
and
then
immediately
VD
gate
can
switch
to
that
replica
and
start
sending
traffic,
and
this
can
happen
literally
within
seconds
and
even
during
those
seconds
what
VT
gate
does
is
it
actually
doesn't
return
errors
to
the
observer
or
it
just
waits
for
the
new
master
to
be
elected.
So
the
only
thing,
the
absolute
notices
is
a
brief
latency
spike
and
the
other
problem
is.
B
Databases
like
my
sickle,
perform
best
when
their
storage
is
local
right,
so
but
but
the
problem
with
kubernetes
is,
if
you
used
a
local
storage,
then
as
soon
as
your
pod
is
shut
down,
kubernetes
wipes
all
your
data,
and
so
in
this
case
that
is
kind
of
scary,
for
anybody
running
on
my
sequel.
So
most
people
that
have
so
far
moved
my
sequel
to
kubernetes
have
ended
up
using
mounted
volume
like
EBS
or
a
CSI
based
volume
that
gives
its
own
persistence
guarantees.
B
There
is
still
now
there
is
now
a
local
PV
that
kubernetes
supports,
but
it
hasn't
seen
much
production
use
yet,
and
it
still
has
a
limitation
that
unless
you're
you're
the
pod
gets
rescheduled
on
the
same
node,
you
still
don't
have
access
to
that.
The
wave
it
has
solves
this
problem
is
it
actually
does
not
rely
on
the
local
data
ever
being
present?
B
B
If
a
pod
goes
down,
you
can
learn
with
us
to
also
use
a
mounted
storage,
and
some
people
do
run
it,
but
but
you
can
also
run
it
in
this
high-performance
high
availability
mode,
where
you
use
local
data
and
then,
if
a
pod
goes,
don't
just
throw
it
away
and
then
bring
up
a
new
part,
let's
storm
replicas
and
keep
moving
forward
and
how
reliable
is
this
methodology?
This
is
how,
with
des
langues
in
google
cloud
in
the
Borg,
we
we
have
never
ever
restored
a
crash
to
my
sequel.
B
We
always
threw
it
away
and
then
brought
up
new
new
instances,
and
this
is
where-
and
this
is
where
running
a
shard
at
my
sequel
makes
more
sense.
If
so,
we
don't
run
huge
instances,
all
our
instances
are
like
two
to
three
hundred
gigs
of
data
size,
which
means
that
if
a
pod
goes
on
a
new
pod
comes
back
up
and
is
ready
to
serve
within
like
a
few
like
five
or
ten
minutes.
So
that's
how
long
it
takes
for
a
new
part
to
be
a
spinner.
B
So
this
is
what
makes
Vitas
cloud
cloud
native
and
the
final
one
is
obviously
the
discovery
mechanism
where
there
is
this
topology,
where
we
do
doublet
when
it
comes
up
it
registers
itself
with
the
topology
and
these
VT
gates
discover
them
as
new
things
come
up
and
last,
but
not
the
least,
it
works
across
multiple
cells.
So
you
can
have
a
full
world
wide
deployment
of
it
as
running
tens
of
thousands
of
nodes,
all
right
moving
forward,
we,
you
can
actually
start
using
with
us
right
from
the
beginning.
B
It
gives
you
benefits,
even
if
you
have
a
single
instance
of
my
sequel,
because
you
gives
you
connection
pooling
it
gives
you
deadlines.
It
makes
overall
your
my
sequel
run
better,
but
then,
as
you
scale,
you
start
to
beg
using
more
and
more
features
of
it.
As
the
Vita's
distributed
system
capabilities,
it
can
route
to
replicas,
it
can
do
load
balance,
it
can
do
master
promotion
and
then
eventually
you
can
char
your
system
and
as
and
as
you
shard
your
system.
B
As
far
as
the
app
is
concerned,
you
still
get
a
unified
view
and
eventually
end
up
into
a
fully
worldwide
deployment.
So
this
is
how
you
would
grow
is
with
us
and,
as
you
can
see,
on
the
left
hand,
side
I
wouldn't
want
to
say
that
app
servers
are
completely
unaffected.
Sometimes
you,
when
you
distribute
things
you
do
have
to
rewrite
things
in
the
app,
because
you
are
making
trade-offs
in
performance
and
not
everything
will
continue
to
work
that
like
it
used
to
work
before,
but
the
impact
is
definitely
minimized.
B
All
right,
so
I
am
going
to
show
a
demo
and
if
I
have
time
at
the
end
of
the
demo,
I
will
talk
about
the
regret
feature,
which
is
when
you
decide
to
shard
a
system.
You
have
to
make
some
trade-offs
and
decisions,
and
sometimes
those
decisions
are
not
the
right
ones.
So
the
latest
feature
that
we
developed
allows
you
to
actually
reach
out
your
system
using
a
different
key,
which
is
why
it's
called
very
good
feature.
So
I
will
talk
about
that
at
the
end.
B
If
I
have
time
so
here's
the
demo
I
am
going
to
show.
This
is
actually
the
use
cases.
Let's
say
we
are
building
a
marketplace
where
we
allow
merchants
to
sell
products
to
customers,
something
like
Amazon
one.
So
this
is
a
simplified
schema
where
you
have
a
product
table
which
has
a
list
of
products,
a
customer
table
where
all
the
customers
are
stored
and
a
merchants
table
that
has
all
the
merchants
and
merchants
list
their
products.
The
customer
comes
in
and
they
say
when
they
place
an
order
or
an
order.
B
Row
is
created,
and
this
order
row
has
foreign
keys,
which
means
that
there
is
an
order
ID,
which
is
its
own
ID.
The
C
ID,
which
is
the
customer
ID
points
back
to
the
customer
that
placed
the
order.
The
PID
is
a
product
that
they
bought
and
the
M
name,
which
is
the
merchant
name,
is
the
merchant
from
which
they
bought
the
product
from
so.
B
If
you
scale
this
massively
you're
going
to
have
billions
of
orders,
you're
going
to
have
maybe
a
billion
customers,
millions
of
merchants
and
thousands
of
thousands
of
products
right
and
then,
when
you
decide
to
short
the
system,
you
are
kind
of
forced
to
split
it
into
three
databases.
One
is
the
product
database,
which
contains
a
product.
One
is
the
customer
database
for
the
customer
and
the
merchant
database
because
they
have
different
keys,
different
route
keys.
B
The
orders
table
is
a
decision
that
you
have
to
make
about
where
to
put
it,
you
can
say
that
orders
can
be
with
merchants
or
or
orders
can
be
with
customer
or
orders
can
be
in
its
own
shot.
If
that
is
a
decision,
that's
also
an
option,
and
each
one
has
a
different
trade
off.
So
if
you
say
that
an
order
can
be
with
customer,
then
you
then
it's
easy
to
join
a
customer
with
order.
You
can
easily
query
whether
you
can
a
customer
can
come
in
and
say.
B
Show
me
all
my
orders
or
you
can
say,
show
me
all
the
orders
for
my
customer.
So
you
can
do
those
kinds
of
queries,
but
it
becomes
harder
to
query.
For
example,
what's
the
product
description
for
a
customer,
because
the
product
description
information
is
in
this
database
or
if
a
merchant
comes
in
and
says
oh
I
need
to
find
out
all
my
orders,
then
it
becomes
a
rather
complicated
query
because
you
have
to
go
to
each
customer
shard
and
find
out
all
basically
essentially
do
a
scan
of
all
shards.
B
So
in
this
case
I'm
going
to
assume
that
the
product
is
uncharted
because
the
product
table
is
small,
a
customer
has
to
be
Charlotte
because
they
are
massive
and
the
merchant
is
also
sharded,
so
I'm
going
to
accelerate
and
show
you
some
pretty
cool
things
we
can
do
with
with
us.
But,
unlike
these
stunts,
these
are
stunts
that
you
can
try
at
home,
so
I'm
going
to
bring
up
a
so
I
already
have
a
cluster
that
I
have
brought
up.
I
can
show
you
the
cluster
here.
B
So
this
cluster
is
the
same
cluster
that
I
showed
in.
In
my
slide,
it
has
one
product
shard,
it
has
two
merchant
shards
and
two
customer
shards.
The
80
means
it's
a
hexadecimal,
so
eight
is
in
the
middle
so
which
means
that
zero
to
eight
is
one
shard
and
eight
to
infinity
is
the
other
shot
so
in
this
and
I
have
an
app
here.
This
is
my
app
my
e-commerce
application,
except
it
is
very
raw.
B
What
you
have
to
do
is
you
have
to
imagine
that
you
are
a
customer
and
when
you
sign
up
you
go
and
execute
the
query
to
insert
yourself
into
this
customer.
If
you
place
an
order,
you
have
to
execute
the
query
to
insert
an
order
into
the
table.
It's
kind
of
a
pretend
application,
but
the
cool
part
of
it
is
you
can
basically
this
app
you
to
send
random
queries
to
this
with
a
shortage
system
and
as
it
executes
it
shows
you
what
it
does
so
just
to
prime
our
I.
B
Have
this.
Let
me
show
you,
for
example,
what
a
B
schema
looks
like.
So
here
is
a
V
schema.
A
V
schema,
basically
is
a
schema,
is
the
description
of
how
a
system
is
charted,
for
example,
for
the
merchant
key
space?
This
is
the
V
schema
where
it
says
that
the
merchant
key
space
has
a
prior
is
charted
by
merchant
name
and
it
uses
in
with
us.
We
have.
B
B
You
can
I
have
some
recoup
statements
that
insert
data
into
this
database
I'm,
going
to
run
my
MySQL
client,
again
sweetie
gate
and
say
import
all
this
data,
and
then,
if
I
go
and
refresh
this
page,
all
my
data
is
imported
in
here
you
can
see
that
there
there
are
different
colors
every
time.
A
new
row
appears
that
row
is
highlighted
by
this.
Axis
is
just
for
convenience.
So
if
I
refresh
this,
everything
will
go
blue.
B
So
here's
some
data
so,
as
you
saw
those
are
just
regular.
My
sequence
is
a
regular
insert
statements,
but
they
went
to
different
shots
like
sugu
went
to
88
ml
went
to
80,
and
but
you
can
see
that
the
orders
for
sugu
are
within
sugus
chart
and
the
orders
for
Demerol
are
within
demo
start.
The
foreign
key
is
the
cid,
which
is
a
customer,
ID,
ID
1
and
in
this
case
ID
6,
which
is
the
demo
shot,
and
now
I
can
run
queries
here
if
I
say
select
star
from
product.
B
So
as
far
as
the
clap
is
concerned,
this
is
one
database,
even
though
it
is
now
split
into
3
into
5
different
basses,
if
I
say
select
star
from
product
what
we
test
us
here.
It
shows
the
executed
queries.
It
says
oh
product.
It
knows
that
it
is
in
the
product
database.
It
says
I'm
going
to
execute
on
product
and
I'm
going
to
send
this
query
if
I
say
select
star
from
customer.
B
That
is
actually
a
scatter
query,
because
customer
is
a
sharded
database.
It
sends
it
to
both
charts
of
the
customers
and
then
sends
you
the
result.
If
I
did
a
join
of
the
customer
with
the
order
table,
you
can
see
that
I'm
saying
customer
join
with
orders
on
the
customer
ID
as
a
foreign
key.
This
query
with
us
knows
that,
since
all
the
orders
are
of
customer
are
within
each
shard,
it
knows
that
it
doesn't
have
to
do
anything
crazy.
B
B
However,
if
I
did
this
query,
so
what
this
query
is
doing
is
is
joining
customer
with
orders
and
then
going
to
product
and
says
fetch
the
product
description
for
each
of
those
orders,
and
please
display
that
right.
So
that
is
something
of
a
cross
short
query.
So
what
we
tests
will
do
I'm
going
to
execute
it
and
then
explain
what
it
did.
B
So
what
we
test
here
does
is
actually
splits
that
query
into
two
parts
and
the
first
query
is
a
join
of
the
customer
with
order
and
as
I
showed
before
it's
a
within
short
query,
so
that
part
of
the
query
runs
as
a
scatter
query,
and
then
it
receives
three
rows
that
came
as
output
and
for
each
of
those
rows.
It
has
to
make
a
round
trip
into
the
product
database
to
fetch
the
description,
which
is
actually
the
three
queries
that
you
see
that
went
into
the
product.
B
So
this
is
so
this
works
in
with
us,
but
it
is
not
optimal,
and
here
with
us
gives
you
an
option.
A
solution
which
actually
no
other
charted
system
gives
that
I
know
of
which
is
this
product
table
is
small.
What
if
we
just
copy
this
product
table
inside
each
of
the
shots
of
this
customer,
then
we
can
tell
the
test
to
say
a
join
with
the
local
product.
Then
this
query
will
be
executed
extremely
efficiently.
So
we
have
this
new
feature
that
is
called
the
materialized
view.
B
You
can
say
materialized
product
in
to
materialize,
any
table
into
any
other
table
using
certain
rules
like
so
I'm,
going
to
give
you
that
materialized
command.
So
here
is
the
command
right,
so
here
I
say
I'm,
going
to
materialize
the
product
dot
product
table
into
customer
dot
product,
and
it
is
a
reference
table.
What
that
means
is
that
I'm
going
to
materialize
all
the
rows
into
each
of
the
shard,
which
means
that
it
is
not
a
Charlotte
table.
B
It's
a
none
Charlotte
table
with
identical
copies
within
each
shard
and
say
I
also
want
the
table
to
be
created,
because
the
table
currently
doesn't
exist.
So
I'm
going
to
execute
this
as
soon
as
I
execute
this
and
then
refresh
it.
The
product
table
has
materialized.
You
can
see
that
it
has
copied
these
rows.
The
one
thing
you
should
remember
is
that
this
table
can
be
multiple
terabytes,
which
means
that
this
materialization
process
can
take
a
while.
It
can
take
a
few
hours,
maybe
even
a
days,
and
during
that
time
this
table
is
not
visible.
B
B
B
Expose
yeah
so
expose
this
so
I
when
I
created
the
materialized
I
named.
This
has
a
workflow
I
use
that
workflow
to
make
changes
to
it
and
they
say
expose
whatever
the
subnetwork
brought
to
the
congregation
now
I'm
going
to
re-execute
the
same
exact
query,
and
now
you
can
see
that
it
is.
The
plan
has
changed.
It
is
now
running
it
as
a
fully
as
a
scatter
query,
where
it
is
joining
with
the
local
product
itself.
B
I
can
still
say
that
I
want
to
specifically
join
with
a
product
table,
and
that
is
possible,
and
so
this
is
so.
This
is
actually
a
very
powerful
feature,
because
things
that
were
extremely
inefficient
can
be
made
super
efficient
by
just
through
much
through
this
material
is
Asian
feature
and
if
and
more
important
thing
is,
if
I
make
changes
to
the
source
tables
now
I'm
going
to
insert
a
new
product
so
as
soon
as
I
insert
it
immediately.
That
row
is
replicated
into
these
target
tables
all
right
so
now.
B
The
next
query
is
what,
if
the
merchant
wants
to
look
at
all
their
orders,
that's
even
harder
query
to
implement,
because
it's
actually
a
full
scatter
on
all
the
merchants
and
for
each
row
that
I
received
it
is
a
full
scatter
on
the
customer
key
space
right.
So
that's
that's
a
query.
That's
basically
unimaginably
expensive
and
again.
The
question
is
what,
if
we
materialize
this
orders
table
into
the
merchants
table.
So
in
the
case
of
a
product
table,
it
was
easy
right.
B
You
just
copied
all
the
rows,
but
in
the
case
of
a
merchants
table
you
can't
just
copy
all
the
rows,
because
one
is
order
is
charted
by
customer
ID
on
and
on
this
side.
For
this
materialisation
to
work
effectively,
it
needs
to
be
shorted
by
M
name,
which
is
actually
the
sharding
key
for
merchants.
So
with
us
allows
you
to
do
that
also,
which
is
essentially
saying
materialized
merchants.
The
customer
orders
into
merchants
orders,
but
when
you
materialize
use
a
different
charting
key
and
also
create
table
right
so
now
I'm
going
to
execute
it.
B
It
feels
like
an
extra
step,
but
in
production
you
need
to
actually
do
this
as
two
steps,
because
you
have
to
wait
for
the
materialization
to
finish,
and
then
you
have
to
expose
it
so
now
that
I've
exposed
it
me
if
I
rerun
that
same
query,
it
now
becomes
a
simple,
a
simple
yeah,
a
simple
cross,
shot
query
in
the
previous
case.
It
was
a
must,
have
multiple
nested
loop
join
very
expensive,
alright.
B
So
if
you
look
at
so,
if
you
look
at
the
underlying
feature
that
helps
you
do,
this
is
called
the
V
replication
feature
the
first
case.
We
actually
copied
all
the
rows.
In
the
second
case,
we
applied
filters
to
these
roles
and
say
it
send
some
rows
here,
send
some
rows
there.
What
else
can
you
do
with
V
replication,
so
V
replication
not
only
allows
you
to
materialize
and
filter.
It
also
allows
you
to
transform.
B
What
does
transform
mean
is
that
you
can
specify
basically
a
select
query
as
an
expression
as
a
transformation,
expression
and,
and
the
Select
query
can
be
actually
can
have
aggregations
and
such,
which
means
that
you
can
actually
create
real
time
roll-ups.
So
here
is
an
example
of
a
real
time,
roll-up.
Well,
what
I'm
going
to
do
is
I'm
going
to
materialize.
B
This
is
again
the
same
materialized
command,
but
here
what
I
have
done
is
instead
of
giving
a
target
table
I
say
it.
I
gave
a
target
expression
saying
that
materialized
sales
as
this
expression,
so
in
this
case
I'm
going
to
select
product
ID
and
the
number
orders
and
the
total
price
for
each
order
and
I
say
materialize
it
as
a
sales
table
and
now
I'm
going
to
also
expose
it
shows
it.
Now.
B
If
I
refresh
it,
there
is
a
sales
table
that
has
popped
up
that
has
automatically
rolled
up
all
the
orders
into
their
counts
and
amounts,
and
then
I
can
now
run
queries
directly
into
the
sales
table.
And,
if
you
actually,
you
can
actually
watch
this
row,
I'm
going
to
change
an
order,
am
I
going
to
yeah,
let's
say
I'm
going
to
add
an
order
here.
If
I
add
an
order,
see
what
happens.
B
The
order
was
added
here
that
added
order
was
replicated
into
the
merchant
key
space
and
the
sales
amount
has
been
updated
accordingly,
with
the
total
of
all
orders,
and
all
this
happens
efficiently,
because
of
how
the
replication
works
is
because
this
happens
by
where
we
replication
subscribes
to
the
bin
logs
and
just
basically
looks
at
each
event
and
applies
just
the
incremental
change
to
the
target
tables.
So
it
is
very
efficient
and
very
fast.
B
The
only
downside
is
that
it
is
eventually
consistent,
which
means
that
if
there
is
a
network
partition
between
customer
and
product,
then
the
sales
table
will
be
delayed
in
its
update.
So
that
is
the
trade
off
for
this
feature,
and
just
before
concluding
the
V
replication
feature
that
we
use
can
also
be
used
for
the
regret
feature
that
I
talked
about.
B
B
A
B
A
good
question
we
test
supports
my
sequel,
five
point:
six,
all
the
way
up
to
8.0.
It
supports
Maria,
DBE,
ten
dot,
X,
all
versions
of
n
dot,
X
all
versions
of
percona
we
test
can
also
run
on
all
cloud
databases
like
RDS
and
cloud
sequel.
So
it's
a
pretty
wide
range
of
the
only
restrictions
it
has
is
that
you
need
to
have
row
based
replication
and
GT
ID
turned
on
so
any
my
sequel
or
mariadb.
That
supports
those
two
features.
It
can
run
on
cachi.
A
B
Yes,
we
have
some
proof
of
concepts
where
we
were
able
to
run
my
sequel
as
well
as
MariaDB
against
rocks,
DB
and,
and
some
people
have
actually
run
it
in
production.
As
far
as
I
know,
there
was
no
Vitesse
limitation
itself.
Those
who
ran
it
on
production
recently
migrated
to
in
ODB,
because
rocks
vb
itself
has
some
limitations.
So
if
my
sequel
with
rocks
DB,
will
work
for
you,
you
can
totally
run
with
us
on
it.
Awesome.
B
Main
difference
is
I
would
say:
scalability
is
definitely
the
biggest
one.
The
prod
Amazon's,
my
sequel,
even
Aurora
beyond
certain
limit
would
not
scale
for
you
as
we
test
you
can
keep
scaling
forever.
Another
big
advantage
is,
even
if
you
can
scale
with
Amazon
with
things
like
RDS
and
Aurora,
you
don't
necessarily
get
to
choose
the
instance
size
that
you
like
right
once
you
grow
beyond
a
certain
size.
You
cannot
really
split
your
database
within
Amazon
itself.
A
B
Totally
totally,
and
and
also
it
with
the
connection
pooling
features,
it
runs
your
mice
equals
much
cooler
and
also
the
biggest
nightmares
of
DBS
is
the
sudden
overload
spikes
and
which
handles
them
really
really
well,
I,
don't
know
if
I'm
allowed
to
talk
about
this
but
I.
There
was
a
recent
outrage
on
slack
and
they
said
we
test
just
didn't
even
blink.
A
B
B
At
the
same
time,
so
the
so
typically
what
I
would
recommend
is,
if
you
are
using
like
if
your
workload
has
a
large
number
of
replicas
anyway,
because
you
have,
you
are
running
a
lot
of
Reap
traffic
and
if
you
are
cross-sell,
because
you
you
need
that,
then
local
storage
is
by
far
a
huge
win.
Not
only
are
you
going
to
get
huge
performance
benefits,
the
fact
that
you
have
these
many
replicas
gives
you
huge
survivability.
B
So
it
is
very
hard
to
lose
the
more
and
more
replicas
you
have
and
the
more
replicas
you
have
cross-sell.
It
is
almost
impossible
to
lose
all
your
data,
whereas
if
you
are
smaller,
where
you
need
only
one
master
and
nothing
else,
you
send
all
your
queries
to
one
master.
In
that
case,
it
is
recommended
that
you
go
with
mounted
storage.
That
is
a
much
more
cost
effective
way
of
running
with
us
right.
A
Right
you
know:
I
I
was
really
intrigued
by
the
whole
kind
of
materialized
sort
of
master
data
distribution
approach.
That's
that's
that's
super
slick
and
especially
the
fact
that
you
can
have
different.
You
know
partition
keys
on
the
data
set.
You
know
to
make
it
you
know
fit
with
the
with
the
model
of
the
the
parent.
B
Correct
they
are,
they
are
super.
They
are
to
be
treated
as
copies,
but
you
could
you
could
go
through
way.
So
the
there
are
two
workflows
that
we
plan
to
support.
One
is
this
materialization
on
almost
identical
workflow,
which
is
actually
a
migration
right,
so,
instead
of
saying
materialized,
you
should
say
migrate.
The
exact
same
thing
happens
in,
but
in
that
case,
what
it
allows
you
to
do
is
actually
cut
over
your
rights
to
the
other
table
right.
A
B
A
B
A
B
It
is
typically
about
two
milliseconds,
so
if
yeah,
so,
if
your
query,
some
people
are,
do
are
sensitive
to
it,
those
that
are
used
to
directly
sending
to
my
sequel
and
my
secure
response
in
millisecond-
and
this
suddenly
is
three
milliseconds,
but
most
people
find
it
absorbable
because
typical,
my
sequel,
latency
is
runs
in
the
five
to
ten
millisecond
range
per
query,
and
in
that
case
this
is
something
that
you
won't
even
notice.
Yeah.
A
A
A
B
B
A
Match
yeah
I
think
there's
like
a
summary
email
that
comes
up
after
and
so
maybe
we
can
drop
it
in
there
get
it
out
to
everybody
that
way.
Maybe
we've
got
time
for
maybe
one
more
question:
there's
a
lot
of
people
interested
in
not
being
got
them
so
excited
about
the
test.
They're
like
hey,
can
I
use
this
with
other
database
engines
like
wins,
Postgres
support
coming
and
you
know
other
databases.
So
can
you
comment
on
that?
So.
B
B
B
A
So
Christmas
we
expected
by
Christmas,
yes,
okay,
cool.
Well,
it
is
top
of
the
hour
again.
So
okay
so
go
super
awesome.
You
know
the
tests
great
stuff
and
thanks
for
the
fantastic
presentation
hope
everybody
enjoyed.
Sorry,
we
didn't
get
to
all
the
questions,
but
I
feel
like
we,
you
know
covered
a
good
swath
of
them,
so
great,
show
and
hope
to
see
you
doing
another
talk
on
the
test
soon.
Thank
you
very
much.
Yeah
take
care,
bye.
Everybody.