►
From YouTube: Vitess and a Kubernetes Operator - Sugu Sougoumarane, YouTube (Intermediate Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Vitess and a Kubernetes Operator - Sugu Sougoumarane, YouTube (Intermediate Skill Level)
Vitess is an open source project started by YouTube to help with scaling of its MySQL databases. It has now gained substantial adoption in the community, especially because it bridges the gap between MySQL and the cloud. This talk will cover the wide range of capabilities Vitess has. We will discuss some case studies of companies that have successfully launched on various Kubernetes environments in GKE and AWS. We will conclude with a demo of the newly-developed Kubernetes Operator for Vitess.
About Sugu
Sugu is CTO at PlanetScale. He is also the lead developer and community leader of the Vitess open source project which he co-created at Youtube in 2010. Vitess has helped multiple companies scale MySQL massively. Prior to Vitess, he worked on various scaling and infrastructure projects at YouTube and at PayPal. Sugu is passionate about distributed systems and consensus algorithms. He publishes his ideas about these topics on his blog at http://ssougou.blogspot.com.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello
good
evening
welcome
to
the
last
and
final
session
of
the
cube
con,
alright,
so,
okay,
how
many
of
you
do
do
not
know
about
the
meta
controllers
by
now?
Everyone
knows.
Oh
there's,
two
people
who
don't
up
do
not
know
IV
I,
like
I
thought,
I'll
be
the
first
one
to
talk
about
it
at
cube
con
and
every
session
that
I
went
to
had
some
mention
about
itself.
Everyone
seems
to
know
what
about
it.
A
A
Databases
were
falling
apart
day
and
night,
and
things
were
completely
out
of
control
myself
on
my
colleague
Mike,
we
said
this
is
not
going
to
work,
so
we
took
ourselves
out
of
all
these
day-to-day
firefighting
and
said,
let's
think
about
what
we
can
do
to
leap
ahead
of
all
these
problems
that
we
are
facing
and
pretty
much
that
solve
it.
This
was
born.
We
thought
this
project
is
going
to
take
about
six
to
twelve
months,
so
eight
years
now
and
not
done
at
all,
so
so
much
so
far,
how
good
we
are
at
estimating.
A
So
the
reality
is
actually
we
launched
we
test
about
a
year
after
we
started
the
project,
but
it
was
so
cool
that
feature
request
just
started
pouring
in
and
we
just
were
busy
adding
more
and
more
and
it
kind
of
organically
grew
into
what
it
became
today,
so
I'm
going
to
start
off
with
and
it
is
also
evolved.
There
are
some
features
that
we
deprecated
some
features
that
we
added,
but
at
this
point
it
solves
three
main
problems:
I,
don't
know
how
many
of
you
we're
at
the
keynote
anywho.
A
So
I
will
summarize
them
here
for
you,
it's
all
scalability
problems
with
your
database.
It
helps
you
move
your
database
to
the
loud
and
it
makes
DBAs
very
happy.
So
those
are
the
three
things
that
it
does
these.
Actually,
these
three
things
are
actually
pretty
related
because
moving
to
the
clouds
make
to
the
cloud
makes
DBS
very
unhappy.
So
this
helps
you
keep
your
DBS
happy.
A
So
this
is
basically
how
it
s
works,
achieves
whatever
I
said
it
does
is
on
the
left
of
the
green
line.
R
is
your
application.
It
was
probably
previously
pointing
to
some
my
sequel
instances.
Now
it
points
to
this
thing
called
the
VT
gate,
which
is
a
retest
component.
A
VT
gate
speaks
my
sequel
protocol.
So,
as
far
as
the
application
is
concerned,
it's
talking
is
still
talking
to
the
same
icicle.
It
was
talking
to
previously,
but
once
the
query
reaches
VT
gate,
a
number
of
things
happen
all
these
components,
VT
gate
and
VT
tablet.
A
They
all
have
an
SQL
parser
and
they
take
a
look
at
your
query
to
decide
what
to
do
with
it
and
they
make
different
decisions
depending
on
which
stage
it
is
in.
So
when
it
comes
to
VT
gate.
Its
job
is
to
figure
out
where
to
send
the
query,
so
it
parses.
Your
query,
looks
at
the
from
clause.
It
says
from
T
where
is
T
or
T
is
in
the
following
database
or
key
space.
A
This
is
what
we
like
to
call
it,
because
in
with
tasks,
we
differentiate
a
database
from
a
key
space,
because
the
key
space
can
be
charted,
whereas
the
database
is
a
single
physical,
my
sequel
instance,
so
it
figures
out
the
key
space.
Oh,
it
needs
to
go
to
that
key
space,
and
it
says,
oh,
that
key
space
is
charted,
which
means
that
I
have
to
figure
out
which
chart
the
row
lives
in.
So
let
me
go
look
at
the
where
clause
where
Clause
says
ID
equal
to
five
Oh,
which
means
ID
equal
to
five.
A
Five
performs
the
magic
and
figures
out.
Oh
it
lives
in
that
chart
and
sends
the
query
there.
So
when
it
sends
a
query,
the
query
reaches
one
of
the
VT
tablets.
So,
as
you
can
see,
there
are
already
here,
there
are
three
types
of
VD
tablets
as
a
master.
There
is
a
replica
and
there
are
already
only
or
what
you
call
as
a
big
data
replicas.
So
that
decision
is
also
made
at
the
VT
gate
level,
where
it
decides.
What
kind
of
query
are
you
running?
Are
you
running
an
update?
Are
you
running
read?
A
Did
you
ask
for
a
dirty
read
so,
depending
on
that
it
decides
whether
that
query
should
go
to
a
master
database
to
a
replica,
or
are
you
running
a
MapReduce
tile,
OLAP
query?
In
that
case
it
sends
it
to
this
replica.
So
the
differentiation
between
regular
replicas
and
a
big
data
replica
is
a
separation
of
workload
because
you
are
allowed
to
abuse
a
big
data
replica
without
affecting
live
production
traffic.
A
So
that's
the
whole
idea
of
this
separation,
so
it
goes
to
the
VD
tablet
and
VT
tablet
says
now:
let's
see
what
we
can
do
with
your
query,
it
doesn't
directly
give
it
to
my
sequel:
it
again
parses
it
and
breaks
it
up
and
decides
what
what
can
I
do
with
this
query
it
may
it
may
do
a
number
of
things
and
I'll
cover
what
they
are.
One
example
is,
it
may
add
a
limit
class
saying
that
this
query
does
not
have
a
limit
clause.
What
if
it
decides
to
scan
the
entire
table
right?
A
What
if
your,
where
clause
is
not
very
good
and
it
scans
the
entire
table,
so
it
adds
a
limit
clause
just
for
protection,
and
if
that
limit
is
exceeded,
it's
going
to
return
an
error.
Instead
of
returning
results,
saying
hey,
you
are
trying
to
fetch
too
many
rows,
so
that's
not
allowed
mainly
for
OLTP
loads.
A
Obviously,
if
you
are
in
the
ole
upload,
you
can
full
do
a
full
table,
scan
no
problem,
and
once
the
VT
tablet
has
modified
your
query
to
make
it
safe,
it
gives
it
to
my
sequel
and
my
sequel:
execute
the
query
returns
the
results
and
the
results
are
returned
back
to
VT
gate.
So
we
ticket
before
sending
the
query
to
VT
tablet.
Might
elf
it
might
have
been
a
query
with
no,
where
clause.
In
that
case,
we
to
gate
might
have
sent
the
query
to
all
shards.
A
If
that
was
the
case,
litigate
collects
all
the
results,
puts
them
together
and
sends
them
back
to
the
app
server.
So
in
reality,
vita
gate
is
designed
to
evolve
into
a
full
multi
shot
database
engine
over
time.
It
is
about
90%
of
the
way
there
right
now
there
are
a
few
parts
still
missing,
but
it
was
written
mainly
to
to
address
most
common
workloads.
A
So
if
your
application,
the
workloads
that
it
doesn't
address,
are
things
like
where
you
shouldn't
be
doing
that
query,
where
you're,
maybe
scanning
a
whole
table
there
and
whole
table
there,
getting
all
the
results
together
and
sorting
them
or
something
if
you
are
doing
crazy
stuff
like
that,
we
ticket
says
sorry
can't.
Do
you
probably
need
to
be
pulling
that
data
in
a
different
form
anyway,
and
how
do
all
these
things
put
together
is
how
does
VT
get
nowhere?
The
VT
tablets
live
and
etc.
The
way
it
happens
is
when
a
VT
tablet
comes
up.
A
It
registers
itself
to
the
read
thing
called
the
lock
server.
The
lock
server
is,
can
be
a
zookeeper
at
CD
or
console,
and
in
google
it
is
chubby
at
YouTube
yeah.
It's
doubly
so,
and
I'll
go
into
the
history
of
why
we
can
support
these
many
log
servers
and
so
VT
tablet
registers
itself
to
the
lock
server
and
a
VT
gate
when
it
comes
up
new.
It
goes
looks
at
the
lock
server.
Tell
me
about
all
the
VT
tablets,
and
then
it
gets
all
the
information
and
connect
to
it.
But
that's
not
the
only
way.
A
Vt
gate
works
with
the
VT
tablet
as
soon
as
it
connects
to
a
VT
tablet.
It
establishes
a
health
check
feedback
from
it
and
then
it
uses
that
help
set
check
to
make
better
decisions
about
which
VD
tablet
to
send
queries
to,
for
example,
replica,
could
be
lagging
what
if
a
replica
is
lagging
way
behind,
we
can
tell
VD
tablet
if
a
replica
is
lagging
beyond
30
seconds,
don't
send
queries
to
it,
so
you
can
specify
rules
like
that
which
VT
VT
gate
receives
directive
from
VT
tablet.
A
Another
important
one
is
when
there
is
a
reparent
when
a
master
goes
down
and
a
new
master
comes
up.
That
communication
is
also
sent
by
the
VT
tablet.
Saying
that
a
I
am
the
new
master,
now
stop
sending
traffic
to
the
old
one.
So
that
helps
the
system
react
immediately
when
there
is
a
reparent
event.
In
most
cases,
zero,
downtime
and
I'll
explain
how,
because
when
that
happens,
VT
gate
when
it
detects
that
every
parent
is
happening.
Vt
gate
just
falls
onto
it.
A
The
queries
just
waits
for
the
new
master
to
come
up
and
as
soon
as
it
comes
up,
it
just
sends
them
all
to
it.
And
finally,
veetc
TLD
on
the
top
left
is
the
dashboard
which
is
used
to
take
a
look
at
what's
happening
in
the
topology.
You
can
also
run
a
few
workflow
commands
through
it.
Alright,
so
VT
VT
tablet
is
what
I
would
describe
as
the
component
that
is
most
responsible
for
making
DBAs
happy.
A
As
a
matter
of
fact,
we
spent
about
four
years,
mostly
working
on
VT
tablet,
the
first
like
2010
to
about
2014,
or
so
it
was
mostly
just
VT
tablet,
and
how
did
we
build
features
into
VT
tablet?
There
is
an
outage,
somebody
gets
woken
up
and
they
complain.
They
say:
hey
I
got
woken
up
for
this.
Why
did
this
happen
and
say?
First
answer
is
always
I
can
happen.
You
know
Utah,
like
is
like.
A
Can
we
get
to
the
root
cause
and
find
out
why
it
happened,
and
is
there
a
way
to
prevent
this
from
happening?
So
over
four
years
we
just
kept
on
adding
things
whenever
there
was
an
outage
either
an
outage
or
it
could
not
be.
It
may
not
be
an
outage.
It
may
just
be
frequent
noisy
pages
right.
If
a
page
is
noisy,
ASR
is
just
hate
it.
They
say
this
Paige
is
noisy
I
need
this
removed
I,
don't
care
what
you're
going
to
do,
but
it's
very
hard
to
do.
A
A
One
person
goes
on
call
and
the
system
is
really
really
quiet.
These
days
have
very,
very
high
QPS
I'm
not
allowed
to
talk
about
how
much
QPS
YouTube
does
and
now
I'm
outside
YouTube
alright.
So
what
are
those
features?
The
first
feature
that
we
did
was
connection
pool.
The
connection
pool
gives
you
two
important
things.
One
is
if
there
is
something
like
a
reparent
event
and
there
are
tens
of
thousands
of
clients
connected
to
the
old
master.
A
They
all
will
instantly
connect
to
the
new
master,
the
new
master,
and
that
usually
means
they
will
destroy
the
next.
My
sequel
instance
guaranteed,
and
this
used
to
happen
so
often
that
we
said
ok,
we
are
going
to
have
a
connection
pool
and
when
a
reparent
happens,
this
connection
pool
will
be
primed,
but
everybody
else
has
to
go
through
this
pool.
So
as
protection
number
one
protection
number
two
is
if
there
is
a
sudden
spike
and
you
send
all
the
queries
to
my
sequel.
A
That
also
is
something
that
my
sequel
does
not
particularly
enjoy,
and
when
that
happens,
this
connection
pool
throttles
the
traffic
to
my
sequel
and
making
sure
that
it's
running
at
maximum
throughput.
So
essentially
by
reducing
how
many
queries
you
send
to
my
sequel,
you
actually
increase
its
throughput.
So
that's
another
thing
that
this
pool
provides
hot
row.
Protection
is
another
very,
very
common
reason
for
things
falling
apart.
A
The
particular
instance
that
I
remember
is
there
was
a
user
who
had
about
250
thousand
videos
or
so
and
they
got
featured
on
the
front
page
of
the
YouTube
index
page.
So,
every
time
somebody
hit
the
front
page
of
YouTube,
250
thousand
videos
were
fetched
from
the
database,
so
imagine
at
whatever
thousands
of
QPS
that
page
was
getting
hit
at
so
that
quickly
took
down
our
database.
So
this
hotter
protection,
what
it
does
is
if
two
identical
queries
hit
the
database,
it
will
send
only
one
are
not
to
n.
A
So
if
n
identical
queries
can
mean
only
one
query
executes
and
once
that
query
executes,
the
results
are
shared
with
all
N.
So
after
this
we
stopped
yelling
at
our
developers
and
at
the
same
time
we
are
also
fulfilling
all
the
queries,
which
is
pretty
awesome,
but
we
don't
know.
The
other
thing
that
it
does
is
the
damage
control
damage.
Control
is,
if
your
query
runs
for
too
long,
it'll
kill
it.
A
So
this
the
the
reason
why
this
feature
is
loved
is
because
it
punishes
the
bad
behavior
the
person
who
wrote
the
bad
query,
otherwise,
I
write
by
it
query.
It
goes
down
and
starts
occupying
all
the
resources
and
brings
down
the
database.
Everybody
pays
for
it
now.
If
I
write
bad
query,
my
query
gets
killed
and
I
get
the
error,
so
I
need
to
go
fix
my
query
before
things
things
can
run
again
for
me
so
there's
these
are
just
some
of
the
highlights.
A
There's
actually
a
whole
bunch
of
features,
and
so
these
are
often
when
people
start
adopting
with
this.
They
come
in
because
they
want
to
shut
their
database
and
stuff,
but
they
get
more
excited
about
these
features
when
they
come
in,
because
these
are
things
that
they
they
faced
along.
The
way
before
they
reached
capacity
and
said,
oh,
even
with
this
I
can
just
buy,
must
buy
some
more
Headroom.
In
fact,
slack.
That's
using
with
tests
they
are,
they
are
in
the
process
of
migration.
A
They
like
these
features
so
much
that
they
actually
customized
a
VT
tablet.
Actually,
you
can
use
it
yourself
as
a
pure,
my
sequel,
proxy,
that
just
sits
in
front
of
my
sequel
that
performs
all
these
functions
for
you.
So
so
that's
a
data
point
so
cloud
the.
How
did
we
test?
Why
is
we
test
cloud
native?
A
There
is
some
history
we
test
was
originally
not
built.
The
cloud
2010
YouTube,
we
were
on
our
own
data
centers
and
this
was
a
bare-metal
product,
as
as
you
would
call
it,
but
in
2013
or
so
snowden
happened
and
google
said
all
data
should
be
inside
google.
So
we
said:
okay,
we'll
migrate,
all
the
data,
and
how
do
you
test
whether
application
can
run
in
the
cloud?
The
same
way
you
test
whether
something
can
float?
You
just
put
it
in
the
water
and
see
what
happens
so,
but
we
test
in
the
water.
A
But
then
we
had
to
make
it
run
and
we
had
even
bigger
challenge
because
we
Tess
was
open
sourced
and
I,
don't
know
how
many
of
you
are
from
Google
or
have
written
in
we
written
code
within
Google.
If
you
write
code
within
Google,
you
cannot
outsource
it
it
because
the
Google
ecosystem
is
so
unique
and
all
proprietary
stuff
it
just
cannot
be
brought
out
into
the
open
into
the
open
source
and
same
is
true.
If
you
have
written
something
into
the
open
source,
how
do
you
change
it
to
you?
A
Can
change
it
to
use
Google,
but
then
you
have
to
abandon
the
open
source.
So
we
had
a
challenge
and
we
had
some
very
heated
conversations.
And
finally,
we
said:
okay,
we
will
bite
the
bullet,
we
will
keep
this
open
source
and
we
will
write
interfaces
for
each
and
every
one
of
these
challenges.
So
we
wrote
interfaces
for
chubbie
interfaces
for
stubby.
We
wrote
interfaces
for
all
these
things
that
are
internal
to
Google,
and
that
is
the
reason
because
we
were
initially
built
on
zookeeper.
We
built
an
interface
for
chubbie.
A
Somebody
said:
can
you
support
HCD?
Oh
no
problem.
Somebody
sat
down
overnight,
we
had
an
HDD
adapter
and
then
somebody
asked
for
console
and
overnight
we
had
a
console
adapter
so
and
same
way
like
we
were
on
beasts
on
our
PC.
Before,
because
we
wrote
that
interface
for
stubby
gee,
our
PC
came
and
overnight
we
were
on
G
our
PC,
so
so
those
adapters
have.
And
finally,
when
kubernetes
came
out,
we
were
overnight
ready
for
kubernetes.
A
A
A
One
thing
is
cloud
like
smaller
instances
and
data
tends
to
be
all
big
like
terabyte
size.
Data
base
is
usually
where
you
start
talking
about
things
right,
so
one
problem
is
shrinking.
That
data
base,
the
other
problem
is
data
bases,
don't
like
to
be
bounced.
They
are
very,
very
unhappy,
even
though
they
are
supposed
to
be
bounce
proof.
They
don't
enjoy
that
kind
of
experience.
So
we
had
to
deal
with
that
and
then
the
other
thing
happened
is
if
you
shrink
these
data
bases
small
and
now
suddenly
you
have
like
a
thousand
of
these
instances.
A
You
cannot
use
the
lock
server
like
he
used
before.
So
one
of
the
experiences
we
had
was
he
just
should
take
down
the
lock
server
because
he
used
to
spam
it
with
calls.
So
you
have
to
reduce
those
calls
and
you
cannot
use
the
lock
server
for
everything
also,
because
if
there
is
a
reparent
event
and
you
want
to
communicate
that
to
VD
gates,
you
cannot
go
through
the
lock
server
because
it
has
delays.
A
So
all
these
things,
there's
are
very
subtle
things
that
you
have
to
kind
of
learn
when
you
move
to
the
cloud
about
how
to
balance
resources,
how
to
assume
the
lives.
The
general
assumption
of
the
life
cycle
of
software
is
very
different,
so
all
those
things
have
to
be
taken
into
account,
and
these
are
things
that
yeah.
So
this
is
the
reason
why
you
are
just
just
being
compatible.
A
I,
don't
know
if
you
remember
what
I
said
in
at
the
keynote
JD
calm
runs
thousands
of
the
test.
Nodes
and
they've
been
doing
it
for
almost
a
year
now,
and
they
are,
they
are
migrating
their
entire
fleet.
They
are
about
25%
of
the
way
right
now,
all
right,
scalability
scalability,
while
the
biggest
feature
of
is
obviously
the
biggest
feature
of
Vitas,
is
that
it
hides
the
pains
of
scalability
away
from
you
by
hiding
all
the
sharding
logic.
So
you
as
an
application.
A
If
you
are
talking
to
an
uncharted
database
or
a
single
database,
you
can
deploy
with
us
on
top
and
point
we
test
with
a
point
to
the
application
to
witness
and
it
continues
to
work
like
before,
but
now
the
effort
you
have
the
opportunity
to
recharge,
and
when
you
do
that,
you
don't
have
to
rewrite
all
of
the
application.
There
will
be
some
impact
because
distributing
data
has
its
cost
and
sometimes
not
all
queries
will
work
the
same
way.
So
some
queries
have
to
be
rewritten.
A
Some
queries
may
not
work
at
all
so,
but
the
impact
is
highly
minimized.
I
would
say
90
to
95%
of
your
application
should
run
mostly
unmodified
and
Vitas
allows
you
to
control
how
sharding
works.
There
is
actually
about
two
or
three
different
ways
you
can
control
charting,
which
actually
has
proven
that
any
type
of
workload
where
sharding
is
needed
it
has,
it
can
satisfy.
I
can
I,
don't
have
much
time
to
go
into
the
details,
but
there
are
talks
where
I've
talked
about
this
one.
There
are
cross
hard
indexes.
So
what
we?
A
What
we
did
was
when
we
decided
to
shard
to
provide
sharding.
We
didn't
go
and
and
use
the
no
sequel
approach.
We
said:
let's
look
at
what
our
database
works
and
extend
those
concepts
into
the
shaadi,
so
we
have
primary
key.
Is
primary
index
or
secondary
indexes?
We
have
schema,
which
we
call
a
V
schema.
We
have
the
notion
on
a
loose
notion
of
a
foreign
key,
which
is
very
useful
to
keep
rows
together
within
the
same
chart.
We
have
sequences
so
essentially
for
you
as
application.
A
You
think
that
you
are
connected
to
a
giant
database,
so
this
is
kind
of
the
roadmap,
even
though
some
inefficient
queries
are
rejected
today
by
Vitesse,
we
think
that
we
should
make
them
work,
because
then
users
can
decide
whether
they
want
to
rewrite
it
instead
of
us
outright
saying
no,
you
can't
use
this
query.
V
schema
base,
V
sharding.
The
way
I
would
call
it.
The
the
the
better
name
for
this
is
called
is
the
regret
feature,
which
means
that
I
decide
to
shard
my
system.
A
A
Synergy
between
Paxos
and
two
PC,
because
they
are
both
two-phase
protocols
and
there
are
ways
to
actually
shadow
one
with
the
other,
so
we
have
some
ideas
about
how
to
do
it,
and
that
is
something
that
we
actually
want
to
do.
I
have
written
some
blog
posts
about
at
hinting
at
how
this
can
be
done.
So
if
you
are
curious
about
these
things,
you
can
read
up
and
or
send
me.
E-Mail
I
can
share
about
what
I
have
in
mind
all
right.
So
with
this
I
am
going
to
jump
to
a
demo.
A
A
Ctl
apply,
manifests
and
then
it's
installed
and
then
once
you've
installed
it
it.
The
way
it
works
is
it's.
It
has
two
parts
on
one
part:
you
define
all
your
CR
DS,
which
is
these
are
all
my
custom
resources
and
and
for
and
then
you
say,
these
are
how
they
are
composed,
and
these
are
the
hooks
that
you
have
to
call
when
they
get
input
they
get
involved.
So
it's
very
simple
you.
Let
me
give
you
an
example:
go
to
beat
this
one
so.
A
Here
so
this
is
the
the
first
part
of
the
controller
which
says
here
is
a
CR
d4,
and
you
can
see
that
the
CR
DS
are
defined
as
things
that
are
with
test
concepts
which
is
like
what
is
the
key
space.
There
is
a
cell,
there
is
a
cluster,
so
these
are
elements
that
are
meaningful
in
the
Vitesse
world.
That
kubernetes
knows
nothing
about
so
these
CR
these
are
defined
here,
and
then
it
says
here
are
how
they
are
composed
is
a
composite
controller,
for
in
this
case
it
is
a
cluster.
A
So
a
cluster
is
composed
hears
and
the
way
you
when
I,
when
I,
ask
you
to
bring
up
a
cluster.
This
is
the
hook
that
you
have
to
run
and
when
I
asked
you
to
bring
up
a
cell
or
change
yourself,
this
is
the
hook
that
you
have
to
run.
So
this
part
gives
you
the
CR,
DS
and
defines
where
which
hooks
to
run,
and
then
the
hooks
are
defined
in.
A
When
you
say
when
you
apply
a
cell,
it
calls
the
sink
cell
loop,
and
this
has
all
the
logic
that
is
needed
to
deploy
a
cell.
It
just
generates
essentially
a
yammer
for
you,
which
the
meta
controller
executes.
So
it's
very
simple.
Otherwise,
writing
a
controller
was
considered
very
heavyweight
and
you
had
to
write
code
and
writing.
Code
is
painful.
This
one,
you
can
write
a
meta
controller,
either
using
javascript
or
json
net,
or
there
are
a
couple
of
other
languages
that
anthony
supports
and
since
Anthony
is
a
Vitas
alumni,
he
decided
to
that.
B
A
A
A
A
My
witness
dot
yeah
Mel,
so
there
is
a
there's,
a
retest
cluster
inside
the
cluster
I
want
some.
There
are
cells,
there
is
a
global
cell.
Global
cell
is
actually
not
a
physical
cell.
It's
actually
the
cell
that
consolidates
everything
and
in
that
I
want
to
run
my
demon.
My
VT
CTL
D,
which
is
my
dashboard,
and
there
is
there-
are
local
cells,
which
are
where
all
the
real
instances
run.
There
is
only
one
zone
here
which
is
own
one,
and
then
there
are
key
spaces,
which
are
those
databases
and
the
each
key
space.
A
There
is
a
name,
and
is
it
chartered
if
it
is
chartered
what
are
the
fraud
ranges
and
which
cells
do
the
key
spaces
run.
So
these
are
all
things
that
are
very
natural
for
somebody
who
is
familiar
with
witness,
and
you
specify
this
and
once
you
specify
this,
you
go
to
your
oh
I
should
show
you
how
meta
controller
deploys
itself
on
the
on
kubernetes.
So
if
you
go
to
the
meta
controller
namespace,
you
can
see
that
in
config
maps
you
see
the
Vitas
operator
hooks.
A
A
There
you
go,
there's
the
Vitesse
operator
running
and
the
meta
controller
running,
so
they
so
the
controller
part
runs
in
a
different
name
spaced,
so
that
it
doesn't
clutter
up
your
your
your
your
namespace
itself,
and
if
you
go
to
the
default
namespace,
which
is
where
things
are,
you
will
now
see
all
the
components
that
I
showed
in
the
architecture.
Diagram.
Let's
see,
was
it
parts
yeah?
A
There
was
it
deployments,
our
deployments
is
only
yeah,
so
in
parts
you
can
see
there
are
there's
a
net
CD
cluster,
the
lock
server
there
is
a.
There
are
VT
tablets
which
are
actually
running
the
my
sequels
and
there
are
there's
a
VT
c
TLD,
which
is
the
dashboard,
and
there
is
a
VT
tablet.
So
if
I
go
to
services,
our
services
is
where
I
meant
to
go.
This
is
this:
has
a
better
high
level
view
of
so
every
box
that
you
saw
in
the
original
you
saw
in
the
original
architecture
diagram.
A
You
can
see
it
here
and
then
I
did
PTC
TLD.
Let's
hope
it's
come
up
now,
and
this
is
the
Vitesse
dashboard.
I
think
it's
coming
up.
Yes,
and
this
says
III
brought
up
key
spaces.
It
has
two
shards.
If
you
go
inside
each
shard,
you
can
see
the
tablets
that
are
present.
You
can
look
at
and
you
have
various
things
that
that
you
play
with
inside
of
it
s.
So
that
is
concludes
my
talk.
Let
me
know
if
you
have
questions.
C
A
Sigma
when
I
say
my
sequel,
it's
Maria,
DB,
percona
or
regular,
my
sequel
or
MySQL,
five,
six,
five,
seven,
eight,
oh
so
with
test,
runs
like
an
application,
so
it
doesn't
really
need,
doesn't
go
into
the
guts
of
my
sequel.
It
drives
it
mostly
like
a
traditional
database
as
a
matter
of
fact.
We
think
they
can
easily
port
it
to
Postgres.
We
just
don't
have
enough
time.
We
have
our
hands
full
handling,
my
sequel
users,
an
existing
schema.
All
you
have
to
do
is
there
are
may
not.
A
Everybody
has
followed
my
advice,
I,
don't
know
why
I
tell
them
if
you
are
running
my
sequel,
just
deploy
with
tests
on
top
and
repoint
your
application,
and
then
you
can
reach
out
it
underneath
a
few
people
did
that,
but
most
people
like
slack,
for
example,
they
said.
Oh,
this
is
a
great
opportunity
for
us
to
rechart,
so
they
actually
deployed
a
Rashard
at
Vitas
on
the
side
and
migrated
the
data
into
their
Rashard
and
and
that's
how
they
are
migrating
there
is.
A
They
are
migrating
into
new
sharted
scheme
than
what
they
had
before
and
some
people,
for
whatever
reason
are
just
copying.
They
did.
They
tie
into
new
my
sequel
lines
and
new
Vitesse
instances
and
without
touching
what
they
are,
what
they
are
serving.
So
all
strategies
have
been
used
at
YouTube.
We
just
deployed
with
tests
on
top
of
existing,
my
sequel
and
because
that
allows
us
to
go
back
and
forth
right.
You
go
send
some
traffic
to
Vitesse.
If
something
breaks
you
fall
back,
so
it
gives
us
that
option
so
yeah
is
this
answer
your
question?
A
A
A
Parsing
we
have
our
own
parser
and
HubSpot
did
a
good
job.
They
had
like
I
think
they
they
they
had
terabytes
of
query
logs.
They
just
dumped
it
on
with
tests
and
saw
what
happened.
Obviously
didn't
many
of
the
queries
failed,
but
they
went
through
and
fixed
most
of
them,
so
it's
it's
still
tries
to
be
the
way
the
parser
works
is
it.
It
tries
to
be
agnostic
when
there's
a
function.
Call,
for
example,
it
doesn't
validate
that
that's
a
valid
function
call
so
it
relies
on
mostly
pass-through.
A
A
Yes,
excellent
question:
yeah
yeah,
very
good
question,
so
the
question
is
how
how
safe
is
with
tests
against
data
loss
so
running
running
in
the
cloud
is
the
scariest
thing
for
people,
especially
they
don't
think
they
can't
accept
that
instances
can
be
ephemeral.
So
how
did
we
make
with
as
robust
against
data
loss?
My
sakura
is
this
feature
called
the
semi
sync
replication.
A
That
gives
you
a
guarantee
that
before
a
commit
is
completed,
the
data
is
replicated
and
sent
to
at
least
one
other
replica
and
so
master
will
not
acknowledge
your
commit
until
actually
will
not
even
commit
your
data
until
at
least
one
other
replica
has
received
it.
If
it
has
received
it,
then
it
commits
and
then
it
returns
and
I
don't
think
this
is
not.
This
is
not
mission
critical
financial,
enterprise-grade
robustness,
but
at
YouTube.
A
We
have
run
this
for
many
years
and
we
haven't
had
an
incident
because
of
this,
and
this
is
also
very
convenient,
because
if
the
master
fails,
there
is
always
a
replica
that
has
the
latest
data,
so
we
can
quickly
identify
which
one
it
is
and
do
a
very,
very
fast
failover.
So
that
has,
and
many
of
the
users
that
can
argue
I
should
be
surprising.
A
Many
of
the
users
that
came
to
use
with
big
reputable
companies
we're
not
even
running
with
this
semi
sync
thing
when
they
say
when
they
realize
oh
wow,
you
get
this
great
resilience
with
semi
sync,
now
most
of
them
actually
with
our
recommendation,
are
running
with
that
mode.
That's
the
answer.
Question.
A
B
A
Yeah
good,
so
the
because
we
use
connection
pools,
you
cannot
use,
you
cannot
authenticate
what
the
user
is
connecting
because
the
user
there
may
be
in
a
company
there
may
be
thousands
of
users
right
or
that
may
be
only
100
users,
so
the
but
the
connection
pool
that
we
test
uses
connects
to
my
sequel
with
a
specific
user
name.
So
the
way
we
work
authentication
is:
oh,
it's
all
done
at
the
Vitesse
level
using
like
if
it's
D
RPC
you
use
certificate
names,
TLS
certificate
names
to
authenticate
that
you
can
call.
A
There's
at
least
one
integration
with
the
LDAP
system,
so
that
authentication
happens
and
that
authentication
goes
all
the
way
to
VT
tablet
and
there
is
actually
a
way
to
configure
ACLs,
for
which
tables
you
can
read,
which
tables
you
can
write
to
is
checked
by
witness
and
accepted
or
rejected.
Based
on
that.
So
basically
we
do
not
rely
on
my
sequels
authorization
or
authentication.
We
test
has
its
own.
B
D
A
They
are,
there
is
actually
future
parity
between
the
two.
The
big
differences
are
Clow
spanner
and
clouds
panel
are
very,
very
consistency,
driven
consistency
and
durability,
driven
which
means
that
with
spanner,
you
will
get.
You
can
get
cross.
Shard
MVCC
reads,
which
is
means
that
you
can
say:
I
want
a
distributed.
Consistent
view
of
the
data
spanner
will
give
you
that,
whereas
we
test
that
drives
individual
databases
doesn't
inherently
support
that
the
other
one
that
spanner
is
is
good,
at
least
with
durability.
A
When
I
talked
about
semi
sink,
it
is
one
other
replica
most
likely
it
could
be.
The
replica
that's
sitting
on
the
next
rack,
or
maybe
even
within
there,
usually
within
the
track.
We
don't
allow
replicas
to
live
on
the
same
black
as
their
master,
so
it'll
be
a
replica
on
the
next
track,
whereas
in
spanner
that
is
extended
using
paxos
across
data
centers.
Well,
of
course,
cap
theorem
is
there.
You
do
pay
the
latency
price
for
that
I'm
using
it
oh
you're,
using
it.
D
A
This
s
gives
you
a
to
PC
completion,
which
means
that
it
gives
you
atomic
transactions.
What
it
doesn't
give
you
is
isolated
views,
isolation
in
views
yeah,
which
is
the
the
way
we
call
it.
Is
you
don't
get
repeatable
read
consistency
you
get
committed,
read
consistency
which
is
when
you
read
whatever
is
committed,
is
what
you
read.
Yeah.
D
D
A
Yes,
we
want
to
so
we
that
is
a.
There
is
an
issue
out
there
that
we
have
to
learn.
We
just
did
a
proof-of-concept
with
RDS
and
not
because
of
any
preference.
It's
because
somebody
was
asking
for
it
and
we
got
with
us
to
run
against
RDS
and
sits
cloud
sequel
is
the
same
type
of
deployment
on
Google.
We
believe
that
it
should
be
pretty
easy
to
get
that
running,
so
we
at
this
point.
We
only
have
a
proof
of
concept,
so.
E
A
Yes,
yes,
definitely
definitely
on.
So
that
is
some.
So
that's
something
that
we
may
do
it
as
planet-scale,
because
people
are
asking
for
it
as
a
paid
service
because
they
are
already
paying
money
to
RDS,
they
say.
Oh,
can
we
like
have
a
a
combined
view
of
it.
So
if
we
don't
but
anything
that
is
outsourced
of
open-source,
will
we,
as
a
matter
of
fact,
actually
the
proof
of
concept
says
that
you
can
run
today
against
cloud
sequel.