►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Vitess - Jiten Vaidya & Dan Kozlowski, PlanetScale
One of the scariest thoughts about storing data in the cloud is the worry about a node going down taking its data with it. In spite of this, many users have successfully run Vitess on Kubernetes in various clouds. This deep dive session in Vitess will show you how to configure a Vitess server to be resilient to failures. As a demo, we'll bring up a Vitess cluster with the a set of optimal configurations and demonstrate how it survives various events like pod evictions and node restarts.
https://sched.co/MPkL
A
A
B
A
A
A
So
you're
not
near
these
mics
yeah,
my
name
is
de
pena.
I
am
the
CEO
of
planet-scale,
which
is
a
company
that
helps
people
run
Windows
clusters.
I
am
the
CEO
of
planet-scale
sugu,
who
is
here,
who
is
the
main
developer
for
Vitesse
and
I,
used
to
work
together
at
YouTube,
where
witness
was
born
to
scale
YouTube's
databases,
I
ran
YouTube's,
asari
and
TB
teams
that
operated
the
Vitesse
clusters
with
me.
A
A
All
right,
so,
let's
start
off
by
talking
about
witness
all
right,
the
witness
architecture.
So
the
reason
that
I
wanted
to
start
out
here
is
that,
first,
let
me
describe
to
you
how
witness
works
and
then
we
can
think
about
how
to
run
it.
A
in
kubernetes
be
in
multiple
kubernetes
clusters
across
multiple
data.
B
A
A
Forth
all
connect
to
an
elf
or
load
balancer,
which
are
front
which
are
backed
by
many
many
instances
of
BTK
process.
Okay,
so
we
TK
to
the
state
this
proxy
that
is
approve
my
sequel
parcel
built
into
it,
and
that
speaks
the
my
sequel,
binary
fellow
to
the
load
balancer.
As
far
as
they
are
concerned,
they
are
connected
to
a
single
humongous,
my
sequel
database
right.
So
maybe.
A
A
A
Depending
on
how
you
are
architecting,
your
environment,
a
cell
is
a
failure
domain
right,
so
in
every
shot,
the
master
and
the
replicas
can
be
distributed
across
multiple
cells.
So
witness
knows
about
cells
from
ground
up
right
and
the
reason
that
witness
knows
about
all
this
and
is
so
good
at
running
in
kubernetes-
is
because
in
Google
it
used
to
run
born,
which
was
the
precursor
to
kubernetes
in
poo
right.
A
B
A
A
You
can
also
send
them
through
a
command
like
multiple
command
line
means,
so
that's
witnessed
in
a
nutshell,
also
want
to
mention.
I
just
want
one.
Second,
that
focus
ever
is
not
part
of
the
query
path
right,
so
topo
sever
has
that
has
the
information
about
how
a
particular
space
is
Charlotte,
as
well
as
the
information
about
the
cluster
topology,
but
all
that
information
is
cached
by
these
v3
gates,
so
that,
even
if
your
proposal
goes
down
your
query
path
doesn't
surface
next
night,
so
wipe
it
s.
A
B
A
Is
a
stateless
entry
point
that
allows
scaling
if
you
suddenly
need
to
add
a
thousand
more
app
servers?
You
increase
your
meeting,
it's
200
or
something
like
that,
and
your
system
scales
right.
There
is
no
connection
number
of
issues
and
so
on
you
can,
depending
on
what
you
want.
You
can
scale
up
and
scale
down
the
number
of
readings.
D
A
Things
like
connection
put
into
the
mice.
Equally,
it
puts
a
timeout
on
every
query.
It
puts
a
timeout
on
every
transaction.
It
has
things
like
hot
draw
protection
built
into
it,
so
in
general
you're.
My
security
is
a
lot
more
robust
running,
underneath
the
supervision
of
within
tablet
than
otherwise,
so
even
for
a
single
non
child
and
my
sequel
cluster,
it
will
perform
a
lot
better
underneath
bit
as
the
other
way.
So
weekly
tablet
is
really
useful.
A
A
Will
kill
you
where
is
that
are
returning
to
many
roles
or
that
everything
for
too
long
right,
I
see
a
lot
of
knowing
smiles
here,
medic
backups
and
restores.
So
when
you
have
multiple
my
signal
clusters,
you
need
some
automation
and
health
around
managing
backups
and
restores,
and
also
a
process
for
building
them
for
bringing
up
new
tablets,
which
doesn't
involve
database
administrator
figuring
out
how
to
bring
up
a
new
tablet
and
so
on.
So
there.
A
Fitness,
which
figures
out
for
I
am
a
tablet.
I
belong
to
this
shard
of
this
key
space.
The
last
backup
was
taken
for
this
shot
and
this
key
space
at
such-and-such
a
time.
This
is
the
s3
bucket
name.
For
that,
let
me
go
get
all
of
that.
Let
me
start
my
my
sequel,
D
and
let
me
start
my
might
not
mind
the
replication
logs
from
the
current
master,
bring
bring
it
up
to
a
certain
threshold
and
start
serving
the
travel.
A
B
A
A
A
B
A
A
You
have
users
in
Japan
hitting
application
servers
in
Japan.
You
can
make
sure
that
the
their
real
queries
are
served
by
the
real
applicants
in
Japan
rather
than
from
the
Masters
in
Mountain
View.
All
of
this
is
again
with
us
at
YouTube,
the
biggest
key
space
that
we
have
was
256
shards,
so
256
pastors
each
master
had
between
80
to
120
replicas,
distributed
over
20
cells
globally.
A
C
On
that
mic,
I
think
this
one's
wrong
hi.
So
gehenna
has
told
you
a
little
bit
about
what
the
global
stateful
kubernetes
app
looks
like,
but
we
wanted
to
show
you
what
a
global
stateful
kubernetes
app
looks
like.
So
what
we've
done
is
we've
set
up
a
little
demo
here.
I
am
gonna,
need
your
help,
so
just
stand
by.
What
do
you
see
here
on
the
screen?
Is
the
architecture
that
I
have
put
together?
C
I
would
have
loved
to
show
it
to
you
live,
but
it
does
take
just
a
little
bit
too
long,
so
we
have
three
different
cells.
Provisioned
two
of
them
are
here
in
Europe.
This
is
all
in
Google
GCP,
so
we
have
one
in
the
EU
West
one
region,
one
in
the
EU
West
four
region
and
then
just
for
kicks
and
giggles
and
because
I
ran
out
of
quota
in
Europe,
we
have
one
in
America,
so
we
have
GU.
C
We
have
the
United
States
East,
one
GU
west,
one
and
E
us,
for
each
of
them
are
running
a
set
of
application
servers
and
a
set
of
attest
shards.
They
are
all
fronted
by
load,
balancers
and
I'm.
Actually
not
sure,
what's
going
on
with
the
ingress,
because
I
know,
Google
has
a
global
cloud
router,
and
it
appears
that
everybody
here
at
this
conference
is
in
fact
hitting
the
EU
4
region
and
that
maybe
Google
shenanigans
I'm,
not
sure.
But
this
is
generally
what
it
looks
like.
So
what
I
need
all
of
you
to
do?
C
Go
ahead
and
grab
your
phones
and
head
out
to
dojo
planet-scale
labs
comm
and
what
you're
gonna
see
is
the
world's
next
big
app
it's
a
little
bit
self-explanatory,
but
I'll
give
you
a
demo
that
you
can
help
me.
What
you're
gonna
be
doing
is
generating
some
load
for
this
cluster.
So
we
can
actually
show
what's
going
on
I'm
gonna
swap
screens
real,
quick.
A
C
C
Okay,
all
right
so
now
we're
back
live
all
right,
so
you
all
should
see
the
goodest
aw
go.
You
should
be
able
to
rate
the
goodest
dog,
oh
okay,
so
here
is
our
nice
little
prometheus
cluster.
Let's
see
how
good
you're
all
doing.
Oh
beautiful,
look
at
that
alright,
so
we've
started
generate
some
traffic
on
there.
C
What
I
want
to
do
is
walk
you
through
a
little
bit
about
the
environment
and
then
show
you
some
common
failures
that
we
have.
So
here
is
the
of
the
test
dashboard.
This
is
again.
This
is
all
part
of
open
source
for
tests.
Here
you
see
that
I
have
two
key
spaces.
One
called
doggers
one
called
lookup
Dockers
is
split
into
four
different
shards.
If
I
go
into
one
of
those
shards,
you
can
see
the
layout
of
the
tablets.
I
have
a
lot
of
read
replicas
here,
mainly
because
I'm
lazy
and
I
was
really
hoping.
A
C
So
right
now
the
master
is
currently
being
served
out
of
EU
s4,
which
again,
that
is
the
region.
I,
believe
that
is
closest
to
Spain.
Here
I
may
have
been
wrong
about
that,
but
what
I'm
also
gonna
do
is
I
also
have
for
demonstration
purposes.
I
have
a
little
script
here.
That
is
just
writing
some
automatic
ratings
into
the
application.
Don't
worry,
they
are
all
very
good
ratings,
so
we're
not
doing
anything
mean
to
these
dogs
and
then
I
have
a
few
other
tablets
to
go
through.
C
What's
going
on
other
things
that
I
want
to
introduce
you
to
while
I'm
here
is.
This
is
called.
This
is
the
VT
gate
interface,
so
the
VT
gate
interface
shows
us
a
little
bit
about
query
traffic.
That's
going
on
again!
We
have
a
number
of
these,
so
the
traffic
that
you
see
here
is
not
necessarily
going
to
match
up
with
the
traffic
across
the
whole
cluster.
To
do
that,
we
rely
on
Prometheus,
which
all
of
the
metrics
are
exported
by
Prometheus.
C
So
we
have
great
visibility
into
not
only
the
queries
coming
in,
but
how
fast
they're
being
serviced
and
I'll.
You
know
there
was
a
little
bit
of
enthusiasm
there
for
a
bit,
but
then
I
guess
people
got
tired
of
reading
dogs,
so
we're
still
at
a
healthy
20
queries
per
second.
So
the
last
thing
I
wanna
introduce
is
this
is
a
tool
called
Orchestrator.
This
isn't
technically
part
of
it,
but
we
integrate
tightly
with
it.
I,
don't
know
how
many
of
you
are
familiar
with
Orchestrator
its
project
by
github.
C
This
integrates
natively
with
the
test,
so
we
can
also
see
here
just
like
in
the
Vitesse
dashboard
I
have
all
of
my
shards.
If
I
go
into
one
of
my
shards
I
can
see
all
my
replicas
and
I
can
see
how
they
are
responding.
Great
great
nice
and
easy
okay,
so
obviously
there's
a
few
things
that
we
may
want
to
have
to
do
with
this
cluster.
C
So
let's
say
that
we
realize
that
we
are
getting
a
lot
of
traffic
in
a
different
region
right
now,
all
the
Masters
like
I,
said
I,
think
all
of
them
are
located
in
Europe
or
maybe
not
all
of
them.
I
have
4
4
4
4,
ok,
so
all
of
my
masters
are
in
Europe,
but
maybe
I'm
getting
a
lot
of
traffic
coming
in
through
another
region
and
I
want
to
optimize
a
little
bit,
but
I
can
actually
do
is
I
can
do
a
planned
reparent.
C
So
I
will
pick
one
of
my
shards
and
I'm
just
gonna
click
on
the
hamburger,
click
on
planned
reparent
and
go
ahead
and
I'm.
Actually
gonna
change
the
master
to
one
that
is
in
sell
us.
West
one
looks
like
two
thousand
and
one
is
in
US
West
one
should
go
ahead
and
pick
it
and
repacked.
It's
gonna
think
about
it
for
a
little
bit
and
it's
gonna
say
good
I've
successfully
repented.
Now,
if
you
are
all
rating
dogs
furiously
you'll
notice,
there
was
no
interruption
traffic
there
and
I
can
go
back
to
my
command
line.
C
App
and
I'm.
Just
chug
chug
chugging
along
the
test,
can
actually
switch
between.
Who
is
the
master
completely
transparently
to
the
applications
using
it.
So
we
can
swap
between
different
regions.
We
can
count
away
from
nodes
if
we
have
to
put
a
node
into
into
maintenance
or
if
we
want
to
cycle
notes
in
and
out,
so
we
can
go
ahead
and
do
that.
So
that's
the
first
fun
thing
that
happens.
The
second
thing
that
happens
is
things
fail
right,
so
obviously
things
fail.
Let's
make
things
fail,
so
keep
CTO
get
all
again.
C
This
is
running
in
three
kubernetes
clusters
across
three
regions.
There's
no
Federation
going
on
here,
because
I,
don't
really
trust,
Federation
and
I.
Think
that's
generally
how
we
all
feel
at
this
point,
so
I
actually
have
three
different
kubernetes
clusters.
Thankfully
the
test
works
really
well.
In
that
situation
we
just
have
a
global
server
and
a
global
Orchestrator
and
then
Vitesse
handles
the
coordination
between
the
three
of
them.
So
I
am
in
u.s.
East
one
I'm
gonna
go
ahead
and
make
sure
I
don't
kill
a
master,
but
I'm
gonna
go
ahead
and
kill
tablet
2003.
B
C
But
I
mean
that's
not
terribly
exciting,
so
we'll
let
that
guy
die
again.
This
could
be
right
now,
I'm
deleting
it.
You
know,
I'm
a
silly
sysadmin
who's,
just
randomly
deleting
stuff,
because
I
don't
know
what
I'm
doing.
But
again,
if
this
was
actually
when
we
were
preparing
for
this
talk,
something
happened
to
my
kubernetes
cluster
node
went
down
tab,
we
went
down,
it
happened
to
be
the
master
and
I'll
show
that
in
a
bit,
but
it
responds
the
same
way
whether
it's
a
hard
failure
or
me
deleting
it.
C
C
So,
what's
going
on,
let's
make
sure
we
can
still
rate
some
poppers.
Oh
you're,
a
good
dog.
You
are
to
13,
so
I,
don't
know
if
you
notice
on
your
phones.
The
scale
actually
goes
to
13,
because
these
are
the
Buddhist
dollars
they're,
not
just
normal,
doggers.
Okay,
so
we're
still
able
to
rate
we
can
see
here
in
BTK,
I'll
go
ahead
and
refresh
it,
and
it's
telling
us
that
some
of
the
replicas
are
down
the
test.
Doesn't
really
you
know
it
lets?
C
You
know
that
it's
now,
but
because
of
those
health
checks
that
your
10
was
talking
about,
we're
not
gonna,
see
queries
fail.
We
were
actually
gonna
see.
The
tests
knows
immediately.
I
mean
within
a
fixed
amount
of
time
that
a
tablet
has
gone
down
and
it
will
in
fact
reroute
all
your
queries
seamlessly.
So
you
know
traffic
chugs
along
normally,
so
there's
that
the
next
thing
that
can
obviously
happen
here
is,
we
could
have
a
master
tablet
fail,
so
master
tablets
are
used
for
all
of
our
rights.
C
So
let's
go
ahead
and
find
where
one
of
the
masters
is.
It
looks
like
the
master
for
the
c0
shard
is
located
at
2001,
so
let's
go
ahead
and
kill
that
one
Orchestrator
is
still
pretty
happy.
It
hasn't
detected
any
failures
which
is
about
right.
Actually,
it
has
detected
failures.
Now
that
it
comes
up.
You
see
those
two
black
dots.
There
was
the
tablet.
Those
are
the
tablets
that
it
thought
were
there
that
are
now
gone
because
I
deleted
them.
So
let's
go
ahead
and
kill
a
master
all
right,
so
I'm
gonna
kill
2001
Boop.
C
Okay,
so
I've
now
unplanned
killed
the
master.
It
is
now
gone
in
a
normal
database
world.
This
is
kind
of
catastrophic
right.
Also,
all
of
these
pods
are
completely
ephemeral.
So
there
there's
no
persistent
volumes
backing
these
up.
They
are
actually
running
on
empty.
What
is
it
empty?
Drive,
empty
disk
storage
so
killing
this
master
here
I
have
actually
killed
everything.
C
We
have
two
ways
that
we
mitigate
against
this
one
is
we
turn
on
something
called
semi
sync,
so
semi
sync
is
a
my
sequel
feature
that
before
we
actually
accept
a
transaction
is
committed,
my
sequel
will
wait
until
at
least
one
replica
has
written
it.
It's
been
log.
Now
it
hasn't
written
it,
it
hasn't
applied
it,
but
it
has
put
it
in
the
bin
log
and
my
secret
will
actually
block
on
that.
So
the
fact
that
I
have
now
killed
this
tablet.
C
Well,
we
should
start
seeing
here
is
some
of
these
inserts
are
starting
to
fail
and
we
can
see
if
I
were
to
just
stop
this
for
a
second,
you
can
see
that
the
c0
shard,
that's
the
master,
I
just
killed.
Some
of
those
inserts
are
down.
Thankfully
Orchestrator
is
there
to
save
the
day,
so
Orchestrator
has
detected
that
there
is
a
failure
and.
C
Because
it's
detected,
there
is
a
failure,
it
is
gonna,
do
something
about
it.
So
Orchestrator
has
recovery.
The
recovery
is
currently
waiting
to
find
out
to
be.
The
successor
is
what
Orchestrator
does
is
like
I
said
with
semi
sink.
You
know
that
that
transaction
was
written
down
to
at
least
one
place.
C
So
what
we
need
to
do
is
we
need
to
find
where
that
transaction
was,
and
that's
the
person
that
needs
to
be
master
so
right
now,
workers
trader
is
actually
evaluating
the
GT
IDs
of
all
the
replicas,
and
it's
gonna
determine
the
one
that
is
most
suitable
to
be
the
new
master,
so
he
does
picked
one.
The
successor
is
ten.
Is
this
host
doesn't
mean
a
lot
to
us
here,
but
we
can
see
it
back
in
the
BT
tablet.
Part
it
is
promoted
the
okay
master
and
it
is
now
going
through
the
rest.
C
Its
gonna
apply
some
changes
to
it.
It
does
take
a
little
bit
of
time
if
I
go
back
here.
What
we
should
see
in
a
few
seconds
is
all
these
queries
start
working
again
and
again.
This
only
affected
that
one
shard.
So
if
you
are
furiously
rating
dogs
right
now,
you
would
have
noticed
that
some
of
those
would
have
returned
errors,
but
generally
it
would
only
need
in
a
quarter
of
requests
and
we're
back
we're
live.
We
have
recovered
from
that
failure.
So
if
and
again
this
could
be
any
number
of
things
that
happened
here.
C
I
happen
to
have
deleted
that
tablet.
But
let's
say
you
had
the
node
go
down
that
was
on
it.
The
back
key
node
unexpectedly
fail.
These
things
happen.
It
would
have
recovered
where,
if
any
other
number
of
unhandled
events
would
have
occurred.
So
if
your
disk
filled
up-
and
you
can
write
to
it
anymore,
if
a
pod
got
evicted,
so
when
one
it's
got
evicted
they
just
they
sit
around
in
the
normal,
require
human
intervention
we
can
Orchestrator
can
handle
that
automatically
for
you,
but
but
wait
there's
more!
C
C
Us
Wes
so
I'm
sorry,
EU
West
for
dot
yeah
mo
just
go
ahead
and
do
that
and
now
I
have
actually
done
something
very
bad.
I
have
deleted
every
single
tablet
and
every
single
shard
and
all
the
VT
gates
that
are
powering
that,
if
you
were
actually
still
furiously
writing
you
may
have
noticed
now
that
the
app
will
no
longer
work.
So
if
you
keep
trying
to
do
it
for
a
little
bit,
you
may
see
it
hang
that's
because
again,
I
pretty
sure
Google
is
routing.
C
All
of
us
to
this
specific
data
center
and
I
have
just
deleted.
Not
only
the
data
base,
but
I've
also
deleted
it
to
be
t-gates.
So
the
database
connection
for
these
app
servers
are
down,
so
I
haven't
actually
been
able
to
figure
out
why
this
is
I.
I
promise
you
and
I
did
this
in
America
I
didn't
get
this
problem.
So
if
you're,
trying
to
rate
Gog's
now
you'll
see
that
that
is
in
fact
failing
and
we
can
see,
the
test
is
very
unhappy.
C
This
is
going
straight
in
through
the
other
European
Data
Center
and
all
of
those
rights
are
failing.
Alright.
So
if
we
go
back
to
Orchestrator,
however,
we
see
that
Orchestrator
has
detected
some
more
failures
right,
so
we
now
have
failures
for
all
of
our
cards.
Almost
and
I
can
go
ahead
and
look
at
recovery,
so
I
see
that
I
have
instantiated
of
recovery
for
c0,
40,
80,
40
and
ADC.
So
this
is
all
of
my
shards
have
now
been
affected
by
this.
Now
you
will
not
be
able
to
commit
anything
until
this
is
fixed.
C
So
again,
Orchestrator
is
doing
in
the
same
song
and
dance
that
it
did
last
time
it
is
going
through
all
of
your
replicas.
It
is
trying
to
which
one
is
regrouping,
replicas
by
a
GT
ID.
So
it's
going
through
all
your
replicas
figuring
out,
which
one
has
semi
synched
all
those
transactions
we
don't
want
to
lose
a
single
rating.
These
dago's
are
not
are
not
something
we
want
to
lightly
brush
aside
once
it
figures
out
who
they
are.
C
It
will
go
ahead
and
recover.
It's
actually
possible
to
do
this
a
little
bit
faster
if
you
are
not
putting
this
in
the
most
suboptimal
situation
again.
I'm
failing
over
between
regions,
so
Orchestrator
actually
has
a
mode
where
it
will
explicitly
disallow
this.
If
you
remember
back
around
six
or
seven
months
ago,
there
was
an
outage
at
github.
That
outage
was
caused
by
Orchestrator,
failing
over
between
data
centers,
which
caused
a
little
bit
of
data
problems
for
them,
because
it
was
tuned
a
little
bit
too
aggressively.
C
So
right
now,
I
do
have
it
turned
on
to
failover
across
data
centers,
because
I
wanted
to
show
taking
down
data
center.
In
reality,
this
may
not
be
how
you
decide
you
want
to
run
it.
Your
needs
may
vary
all
that
aside.
Let's
see
how
it's
doing
okay,
so
it
has
picked
a
master,
it
has
picked
a
successor
and
it
is
starting
to
apply
changes.
I'm
gonna
go
back
to
VT
VT,
it's
gonna
be
very
red.
Right
now,
ya
see
its
unhappy,
it
is.
C
Okay,
so
it's
picked
the
master
it's
now
going
through
the
recovery
procedure
on
that
master.
Also,
this
is
not
a
global
Orchestrator,
I
kind
of
cheated
here
and
I
only
put
worker
Schrader
in
one
data
center.
So
all
of
this
recovery
is
actually
happening
across
data
centers
as
well.
Orchestrator
has
a
mechanism
that
you
can
run
as
a
a
raft
consensus
group,
which
is
how
we
would
run
this
in
production
and
how
we
set
it
up
the
planet
scale
when
we
run
the
test
for
people
okay.
So
now
it's
done
its
thing.
C
It
has
successfully
tablet
externally
parented
and
we
can
sort
of
see
here
in
VT
gate
that
it
has
picked
a
new
set
of
masters
for
at
least
three
of
the
four
no
ball
four,
so
all
four
of
them
now
have
one.
One
of
them
ended
up
in
America,
go
figure.
If
I
go
back
to
my
terminal,
we
have
resumed
inserting
data.
So
just
like
that,
you
lose
a
data
center.
We
did
have
about
two
minutes
of
outage
here.
C
In
our
experience,
we
can
get
that
down
a
little
lower
I've
been
able
to
get
fail,
overs
in
as
little
of
nine
seconds.
But
again,
if
you
tune
this
too
quickly,
you
can
get
into
a
situation
where
you
failover
into
something
you
don't
want
so
and
then,
if
I
go
back
to
my
good-ass
dogger
up
up
the
google
shenanigans
continue.
C
Okay,
well,
I
am
NOT
a
very
good
web
developer.
I
imagine
this
is
my
fault
anyway.
The
inserts
have
resumed
again.
Your
database
has
recovered
again.
You
can
do
this
without
any
human
intervention.
So
all
the
stuff
that
you
saw
me
do
here.
None
of
this
was
actually
to
recover.
The
app
all
I
did
was
delete
lots
of
things.
We've
actually
heard.
Some
people
who
use
with
tests
have
had
similar
situations
happen
where
they
have
lost
enough
nodes
that
they've
lost
the
majority
of
the
Masters
on
their
database
deployment
Orchestrator
kicks
in
rhe
parents,
everything.
C
That
being
said,
it's
not
perfect,
there's,
obviously
situations
that
still
require
human
intervention,
so
you
can't
throw
away
your
pagers
just
yet,
but
this
is
some
of
the
stuff
that
you
can
do.
I
think
we're
running
out
of
time
here
and
I
did
want
to
leave
some
moments
for
questions,
so
we're
going
to
go
ahead
and
present
again
I
gonna
blow
through
the
next
couple,
slides
like
I,
said
they're
ephemeral,
so
we
are
restoring
from
backups
and
then
catching
up
by
a
replication.
C
We
are
not
using
persistent
disks
anywhere
in
this
there's
a
few
places
where
you
have
less
common
problems
that
do
require
you
to
actually
delete
things
or
interact
with
the
cluster.
We
have
some
catastrophic
failures
like
unplanned
node
failures.
That's
when
I
believe
the
one
master
or
a
multi
region
data
center
outage.
So
right
now
like
when
I
deleted
that
whole
region,
that
was
three
data
centers,
it's
really
unlikely
for
Google
Cloud
to
lose
an
entire
region
like
that,
but
even
if
it
does,
if
you're
using
the
tests,
your
database
will
recover.
C
So
the
last
thing
is
the
test
is
a
CN
CF
project.
It
is
open
source.
We
would
love
for
you
to
join
our
community
to
contribute
to
the
project
if
you
like
databases
or
are
interested
in
computed
systems,
please
stop
by
its
of
a
test
on
Io.
If
you
want
to
join
us
on
slack,
that's
really
where
we
do
most
of
our
collaboration.
If
you
go
to
the
test,
dot,
slack
comm
or
if
you
go
to
fit
test
at
I/o,
you
can
invite
yourself
to
our
slack.
C
It
is
found
on
github
if
you
really
like
the
test,
and
you
want
to
work
on
it.
Full-Time
the
team
at
planet-scale
is
hiring
and
we
would
love
to
talk
to
you.
So
please
either
email
us
there
or
come
find
one
of
us.
So
we
do
have
a
few
more
minutes.
I
hope
ya,
good.
Nobody
said!
No,
so
we
do
so.
If
anybody
has
any
questions
about
the
tests
or
about
what
you
saw
or
about
where
I
got
all
those
adorable
pictures
of
dogs.
You
just
let
me
know.
C
Okay,
so
the
question
was
the
question:
was
about
semi
sink
replication?
How?
If,
if
I
wait
for
an
acknowledgment
from
one
replica
before
I
say
it's
okay,
what
happens
if
I
lose
the
master
and
that
replica
that's
a
great
question.
You
have
two
options
in
that
case:
one
is,
it
doesn't
have
to
be
one
you
can
set
that
number
to
be
whatever
you
want.
So
if
you
want
it
to
be
two
replicas
you
can.
The
second
thing
you
can
do
is
obviously
in
this
situation,
where
I
have
co-located
replicas
and
masters
right.
C
The
place
where
that
the
place
where
semi
sink
triggered
the
acceptance
is
going
to
be
the
local
one
right.
Just
the
laws
of
physics
dictate
that
most
of
the
time
it's
going
to
be
the
one
that's
closest
when
I
lost
that
data
center
I
lost
both
of
those
nodes.
The
way
that
we
have
around,
that
is,
you
don't
have
to
turn
on
semi
sink
for
all
your
nodes.
The
only
nodes
that
participate
in
semi
sink
are
the
ones
that
you
explicitly
enable
it.
We
let
talk
about
slack
yeah,.
C
C
So
you
are
not
fixed
to
a
given
version.
I
think
there
are
some
okay.
The
question
was:
are
you
fixed
with
a
specific
version,
or
are
you
limited
to
only
certain
features?
No
you're?
Not
so
we
have
compatibility
from
my
sequel,
five,
six
up
to
my
sequel,
eight
Oh.
That
being
said,
it
is
the
test
does
have
its
own
sequel
parser
in
it.
C
So
there
are
some
things,
especially
in
my
sequel,
eight
that
we
haven't
implemented
yet
so
I
know
we
have
not
implemented
the
JSON
JSON
portions
of
my
sequel,
eight,
so
that
if
you
were
try
to
issue
a
query
in
a
multi
sharded
environment
with
JSON,
we
wouldn't
be
able
to
do
that.
For
you.
If
it
was
symbol
sharted,
we
would
still
be
able
to
pass
it
right
there
and
whatever
you're
backing
version
of
my
sequel
was
in
English,
art
and
environment.
We
will
handle
all
the
features.
Also
in
a
sharded
environment.
C
A
B
C
So
the
sharding
is
built
into
the
test.
It
is
the
one
of
the
primary
features
of
a
test,
so
there
is
no
other
technology
that
does
it
to
the
outside
application.
The
test
looks
like
a
single
database.
In
fact,
you
can
start
to
deploy
the
tests
by
deploying
it.
If
you
have,
if
you're
using
my
sequel
today
and
you're
running
out
of
capacity
on
my
sequel,
you
can
actually
put
the
tests
in
front
of
it.
C
It
can
operate
and
then,
with
with
what
we
call
unmanaged
my
sequel
mode
and
it'll
just
point
to
your
existing,
my
sequel
server
queries
can
come
through
the
tests
and
they
can
be
serviced
by
your
existing,
my
sequel,
server
and
then
returned
in
for
reference
the.
So
there
is
no
knowledge
of
a
test
built
into
the
application
itself.
Here
is
good
as
doggo.
This
is
completely
planet-scale
proprietary.
None
of
you
are
allowed
to
mention
that
you
saw
this
so
here
as
good
as
they'll.
Go,
it's
not
a
very
complex
app.
C
You
can
see
that
there
is
just
three
standard
queries.
This
would
work
the
same.
Whether
I
had
my
sequel
behind
it
orbit
test
behind
it.
In
fact,
this
just
uses
the
community,
my
sequel,
drivers
for
rust
and
it
connected
right
to
Vitesse.
So
in
terms
of
performance,
which
I
think
was
your
last
part,
there
is
a
performance
overhead
freezing
to
tests.
C
We
found,
if
you're,
doing
benchmarking
against
that,
and
you
have
sub
sub
millisecond
queries,
then
that
doesn't
reflect
well
on
the
tests
under
normal
operations,
where
you
maybe
have
10
or
15
millisecond
queries,
we
don't
really
notice
it
that
much.
The
second
thing
that
you'll
see
is
that
we
use
anywhere
from
20
to
40
percent
CPU
20
to
40%
of
the
CPU
that
you're
my
sequel
process
will
use,
and
that
really
also
depends
on
the
query
workload.
C
So
in
general,
the
tests
to
my
sequel
you'll
probably
see
anywheres
from
75
to
90%
performance
as
to
what
you
would
get
over
straight.
My
sequel,
however,
as
soon
as
you
shard
that
you're
now
way
above
what
a
single
my
sequel
server
could
do
for
you
anything.
You
wanted
add
that
and
then
you
wanted.
Oh
okay.
D
C
Under
under
a
lot
of
circumstances,
we
do
pause
and
I
was
actually
I
could
have
sworn
to
you
before.
We
did
this
that
we
actually
paused
on
these
failures
as
well.
The
test
has
a
built-in
buffer
for
situations
like
this.
So
whenever
we're
doing
things
like
restarting
re
parenting,
so
you'll
notice
money
to
the
reparent
we
switched
masters.
C
The
test
knows
how
to
buffer
queries
so
that
we
safely
handle
that
reparent
teen
will
actually
will
buffer
all
begins
will
allow
all
commits
to
finish,
and
then
we
will
swap
the
masters
and
then
we
will
allow
the
begins
to
continue.
So
it
has
that
buffering
built-in.
That
is
a
great
question
as
to
why
we
don't
do
it
when
the
backing
tablet
goes
down
and
it's
definitely
something
that
we
are
now
going
to
actively
try
to
convince
sugu
is
is
absolutely
necessary.
C
D
C
The
GT
IDs
are
gonna,
be
through
replication.
They're
gonna
be
replicated
to
all
the
replicas
in
that
replica.
Set
that
that
cluster
so
and
that's
the
whole
in
the
magic
of
semi
sync
is
we're.
Gonna
make
sure
that
a
specific
GT
ID
has
made
it
to
at
least
one
other
node
or
a
configure
amount
of
nodes
to
make
sure
that
it's
not
a
single
point
of
failure.
C
B
C
I,
don't
actually
know
what
it
is.
Yes,
it
does.
In
fact,
we
we've
worked
extensively
with
Amazon
to
get
people
running
the
test
on
top
of
RDS
we've
partnered
with
Amazon
for
their
aurora
product
to
show
just
how
wide
there
were
just
how
performant
their
aurora
product
is.
We
haven't
published
that
blog
post
yet,
but
it's
it's
gonna
be
fantastic.
When
we
do
so,
yes,
it
works
perfectly
there.
There
are
some
things
that
you
can't
do
or
that
you
don't
do
in
Vitesse
if
you're
using
aurora.
C
So
if
you
are
using
aurora,
most
people
would
want
to
have
World
War
manage
the
master
and
replicas.
So
in
Vitesse
you
normally
would
only
when
we've
deployed
into
test,
we
only
put
up
two
tablets,
no
matter
how
many
actual
physical
Aurora
hosts
or
RDS
hosts.
You
have.
We
only
put
up
two
tablets
and
then
we
we
do
it
that
way
and
we
allow
order
to
scale
the
reads
instead
of
in
this
case,
I
had
a
bunch
of
read
replicas
and
they
could
each
they
managed
by
the
test
and
it
could
choose
between
them.
C
So
discover
is
the
devil
it
does
so
we
can't
use
discovery,
and
so
there
is
a
orchestrated
configuration.
It
does
need
a
couple
extra
things.
Besides
the
standard
one,
there's
a
couple
posts,
failure
hooks
that
we
have
to
add
specific
for
tests.
If
you
look
on
the
Vitesse
github
page,
the
test
has
a
helmet
art,
that's
developed
by
the
community
that
has
an
Orchestrator
config
in
it.
That
Orchestrator
config
is
mostly
correct.
C
It
will
generally
work
for
you
if
you
were
on
the
Vitesse
slack
channel
you'll
notice
that
recently
some
members
of
the
community
have
had
some
hard
times
with
that.
So
both
preparing
for
this
and
then
generally
because
we
want
to
see
the
test
succeed,
we
have
a
modified
version
that
you're
gonna
see
committed,
probably
when
I
get
back
to
the
US.
That
is
very
reliable.
A
C
Alright,
well,
we
are
I
believe
over
our
time,
but
thanks
everybody
for
coming
and
watching.
If
you
have
questions,
please
let
me
know
if
you
want
to
keep
raiding
dogs
because
I
know
it's
super
addictive
I'm
gonna
leave
that
app
up
a
little
bit
and
let's
see
if
it
came
back
online,
you
know
what
I
bet.
It's
the
whole
thing
with
Spain.
So.