►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Vitess - Dan Kozlowski & Sugu Sougoumarane, PlanetScale
Vitess has been a member of CNCF since early 2018, and is continuing to gain adoption as the Cloud-Native database solution. In this session, we are going show how Vitess can be used to never require a maintenance window.
To learn more: https://sched.co/Grd9
A
And
yes,
and
the
co-founder
of
the
Tess,
so
I
have
labeled
this
presentation.
Failure
is
always
an
option
and
I
am
going
to
run
completely
out
of
time,
because
I
got
started
late
and
I
have
way
too
much
to
cover.
So
one
of
the
things
I
love
about
the
tests,
I
always
say
it's
normally
a
cloud
native
database,
it's
the
first
cloud
native
beta
database
and
whenever
you're
operating
in
a
cloud
native
fashion,
you
have
to
plan
for
things
to
fail.
So
in
a
perfect
world
we
don't
care
about
our
machines.
A
We
don't
care
about
their
connections.
What
we
really
care
about
is
the
service,
so
with
that,
what
I
wanted
to
do
is
go
in-depth
a
little
bit
about
the
different
ways
that
things
with
Vitesse
can
fail
and
how
we
recover
them.
So
just
things
that
you
already
know
right
failure
is
part
of
life
machines.
Go
down
all
the
time.
A
There's
two
things
you
can
do
when
something
fails
you
can
handle
it
or
you
can
just
pass
it
on
down
the
line
for
somebody
else
to
handle
it
for
Vitesse.
We've
actually
put
a
lot
of
things
into
the
tests
that
handle
the
failures
at
the
database
level.
So
if
you
have
a
node
go
down
or
if
you
have
a
whole
data
center
go
down
that
doesn't
necessarily
get
propagated
up
to
your
the
clients
downstream.
A
So
you
decided
that
you
would
like
to
have
a
cloud.
Alright,
you've
decided
you'd
like
to
have
a
cloud
native
database
that
can
survive
failure.
There's
a
few
things
that
you
have
to
think
about
before
you
even
begin
this.
The
first
is
where
you're
gonna
place
things
to
handle
them,
doesn't
matter
what
system
you're
running
right.
A
The
second
thing
you
have
to
know
how
to
do
is
you
have
to
know
how
to
recover,
so
we
can
plan
for
the
different
kinds
of
failures
we're
gonna
have,
but
then
there's
got
to
be
a
way
to
get
out
of
it,
and
then
the
third
and
probably
the
most
important
thing
is
to
know
what
kind
of
failure
you
can,
how
much
risk
you
can
accept.
So
things
are
gonna
fail
in
a
lot
of
cases.
There
is
gonna,
be
some
kind
of
outage.
There's
gonna
be
some
kind
of
noticeable
outside
event.
That
happens.
A
A
So
when
we
built
the
test
to
handle
failures,
what
do
we
do?
The
first
thing
that
we
do
is
we
separated
the
components
so
in
a
normal
database
system,
you
have
a
single
node
that
single
node
has
an
endpoint.
If
it's
my
sequel,
it's
3306,
you
listen
to
3300
on
3306
and
when
somebody
connects
you
serve
that
data,
you
go
all
the
way
down
to
disk
or
to
remote
disc,
to
get
the
data
and
send
it
out
in
the
test.
A
We
actually
have
three-tier
architecture
and
since
this
is
a
deep
dive,
we're
not
going
into
it
because
I
hope
everybody
here
knows
what
it
looks
like
you
can
see
me
afterwards.
I'll
show
you
the
architecture
diagram,
but
there
is
a
gateway
process.
There
is
a
handler
for
the
database
and
then
there's
a
database
itself.
By
separating
out
these
components,
we
can
handle
the
failure
at
any
single
tier
and
be
able
to
prop
handle
that
failure
at
the
next
tier
without
people
knowing
about
it.
A
In
fact,
if
you
are
using
the
G
RPC
drivers
to
connect
to
VT
gate,
you
actually
can
propagate
handling
that
failure
all
the
way
out
to
your
client
application,
and
you
can
even
survive
a
VT
gate
going
down.
That's
what
I
mean
when
I
say
push
state
to
the
edges,
so
we
push
it
out
to
the
edge
with
VT
gate.
We
push
it
out
to
the
other
edge
with
multiple
mice
equals
all
right.
So
to
really
show
this
I've
created
a
sample
cluster
for
you.
This
sample
cluster
is
running
in
two
Cooper
Nettie's
kubernetes
clusters.
A
It
is
running
on
Google
Cloud.
One
of
them
is
located
in
u.s.
West
one.
One
of
them
is
located
in
u.s.
East
one,
which
is
not
what
that
slide
says,
but
that's
where
it's
at
in
these
two
Cooper
naty's
clusters
I
have
two
key
spaces.
One
of
them
is
a
lookup
key
space,
which
the
only
thing
in
there
is
two
sequences.
The
other
is
the
actual
database
for
my
application.
The
application
that
we're
gonna
demo
is
something
that
I
call
goodest
doggo
or
goodest
poppers.
So
this
is
what
it
looks
like.
Here's,
the
goodest
augo.
A
If
you
go
to
popper,
that
planet-scale
labs.com
you'll
be
able
to
play
along
and
rate
the
dogs
right
now,
it's
not
on
because
we're
gonna
turn
it
on
together.
This
is
the
application
and
what
we're
gonna
show
is.
We
can
lose
any
point.
We
can
lose
nodes,
we
can
lose
pods
and
we
can
lose
an
entire
data
center
and
still
have
this
application
work
a
little
bit
about.
A
What's
on
the
periphery
there
we
also
have
VTC
TLD,
which
is
our
control,
plane
and
VT
Gate,
which
was
our
gateway
running
in
both
of
these
pods
and
that
URL
that
I
told
you
puppers
dot,
planet-scale
labs
com,
that's
actually
a
DNS
load-balanced
lookup
between
these
two
regions.
So
if
you
go
there
theoretically,
half
of
you
will
get
us
east
and
half
of
you
will
get
us
west.
Okay.
So
let's
talk
about
failure
and
recovery
here
are
the
different
things
that
can
fail
and
the
impact
it
has
on
your
system.
A
Vtc
TLD
can
fail
and
absolutely
nothing.
No
one
cares
right.
Vt
c
TLD
is
not
in
the
serving
path
so
that
guy
can
go
down
and
come
back
as
much
as
he
wants.
The
only
thing
that
can't
happen
is
you
can't
make
cluster
changes
when
VT
c
TLD
goes
down
so
from
an
operational
standpoint,
not
the
best,
but
it's
not
gonna.
Stop
you
from
serving
traffic.
The
second
thing
that
can
go
down
is
VT
gates.
Vt
gates
are
the
gateway
processes.
That
is
what
your
application
servers
connect
to
for
this
demo.
A
A
It's
actually
not
that
big
of
a
problem,
cuz
clients
will
reconnect
they'll
come
up
with
a
different
VT
gate
and
then
they
will
start
going.
But
that
is
the
impact
on
losing
a
VT
gate.
Next
is
VT
tablets.
Vt
tablets
are
the
Wranglers
for
my
sequel.
If
VT
tablet
goes
down,
the
database
probably
should
that
shard
is
not
going
to
report
as
being
active
and
you're
not
gonna,
be
able
to
use
it
so
VT
tablet
and
my
sequel
are
roughly
the
same.
A
It's
a
little
bit
different,
how
we
recover
from
them,
but
with
VT
tablet
transactions
will
fail
with
my
sequel:
you'll
have
to
recover
okay.
So,
let's
go
take
a
look
at
where
we're
at
right.
Now
we
have
the
system
that
I
just
described.
This
is
the
live,
VT
c
TLD
tablet
interface
I
have
my
lookup
shard,
which
is
not
charted.
As
you
can
see,
there
are
some
nodes
and
uswest
some
nodes
in
u.s.
East
and
we
have
the
double
shard.
Dago's
are
also
in
u.s.
West
and
US
East.
A
Two
of
these
shards
are
actually
have
masters
in
u.s.
West
and
two
of
them
have
masters
in
u.s.
East.
Okay,
if
at
any
point
someone
has
questions,
feel
free
to
ask,
however,
we
can
see
for
various
reasons.
Some
of
them
are
not
happy
so
in
VT
gate.
This
is
the
VT
gate
interface
for
those
of
you
who
haven't
seen
before
this
is
our
view
into
the
Vitesse
world.
It
shows
us
how
things
are
going.
We
can
see
that
a
couple
of
these
replicas
have
high
replicas
lag
so
the
way
that
we
recover
from
replicas
lag.
A
The
reason
is
because
just
before
I
walked
in
this
door,
I
actually
killed
a
bunch
of
pots.
So
you
can
see
all
of
those
pots
now
have
replicas
lag.
What
I'm
gonna
do
is
I'm
going
to
go
back
to
VT
gate
or
VT,
see
TLD
and
I'm
going
to
pick
my
shards
that
have
replicas
lag
and
I
can
do
two
things
if
you're
in
VT
c
TLD.
So
this
is
gonna
go
through
a
lot
of
the
manual
process
of
recovering
clusters
by
the
way.
A
So,
if
you
set
up
a
of
a
test
cluster,
this
is
what
you'll
get
a
couple
of
words.
I'm
gonna
say
is
I'm
gonna,
say
hamburger
a
lot,
these
three
two
lines:
they're
the
hamburgers.
So
if
you
go
to
VT
CTL,
D
and
click
on
the
top
hamburger
here,
you
will
get
the
pop-up.
That
shows
you
all
the
actions
you
can
do
to
recover.
V
ccTLD,
the
ones
that
I
am
most
concerned
with,
are
always
planned.
Reparent
emergency
of
your
parent
and
externally
rhe
parented.
A
A
So
what
I've
effectively
done
is
I've
used
Vitesse
to
go
into
that
my
sequel
instance
and
issue
the
same
commands
that
you
would
run
if
you
or
a
DBA
and
you
were
running
into
it.
So
if
I
refresh
this
now,
we
can
see
that
the
replication
lag
has
gone
to
zero.
So
let's
go
do
it
for
the
other
ones,
1102.
A
A
A
So
now
we
can
see
that
almost
all
of
the
errors
have
gone
away
with
the
exception
of
one
three
zero
zero
one
I
am
just
gonna
keep
soldiering
along,
though
the
next
thing
that
I
want
to
do
is
I
actually
want
to
bring
up
the
service
or
I
would
like
to
load
the
service.
So
now
that
we
have
the
cluster
online
in
a
healthy
state,
let's
go
ahead
and
load
some
of
the
some
of
the
sequel
statements
for
it
so
hold
on
a
second.
A
Okay,
these
schema
has
been
applied
schema.
Let
me
just
walk
you
through
it.
You
have
the
V
scheme,
which
tells
you
how
we're
gonna
shard
our
tables,
we're
gonna
char
them
by
name
and
popper
name.
There
are
the
same
value,
so
they
will
shard
equally
and
then
I
have
two
sequences
and
my
lookup
table
that
way.
I
can
have
sequential
primary
keys
for
my
own
ODB
tables.
At
this
point,
I
have
a
fully
working
my
sequel.
A
A
All
right,
so
we
have
a
database.
We
have
the
database
up
and
running.
What
can
we
do
to
start
to
harass
the
database?
So
what
I'm
gonna
do
is
start
killing
a
few
things
again.
This
is
running
in
kubernetes
and
what's
gonna
happen
is
when
I
kill
these
pods
we're
running
in
a
persistent
fashion.
They
have
jeast,
they
have
the
Google
persistent
disks
behind
them.
So
after
we
kill
them,
they
will
come
back
up
and
I
can
then
show
you
how
to
recover
from
it.
So.
A
A
A
A
A
A
A
A
Okay,
so
again,
failure
of
given
nodes
when
you
fail
a
specific
node
out
of
your
cluster,
especially
in
fronting
in
kubernetes.
The
results
are
going
to
vary
I
just
deleted.
All
the
pods,
but
I
haven't
deleted
any
of
the
nodes.
Deleting
a
node
is
gonna,
determine
which
services
go
down
when
you
lose
a
replica
in
the
test,
there
are
replicas
and
there
are
masters
when
you
lose
a
replica.
A
The
response
is
to
do
something
called
a
plan,
debris
parent
or
an
emergency
reparent
if
it's
an
unknown
loss
and
that
will
shift
your
traffic
from
one
master
to
another
master.
So
if
a
node
goes
down
and
it's
a
replica
node,
you
don't
have
to
do
anything
because
replicas
aren't
in
unless
it's
a
semi
sync
state,
with
only
one
replicas
you're,
not
gonna,
be
in
a
place
where
you
have
cluster
unavailability.
If
the
node
goes
down
that
is
holding
the
master,
you
need
to
then
do
a
planned
reparent
or
an
emergency
repair
it
depending
on.
A
If
you
know
the
node
is
going
down
or
if
you
don't,
the
other
thing
that
can
go
down
is
the
entire
data
center
can
go
down.
So
if
a
test,
we
have
the
concept
of
cells,
you
can
think
broadly
of
a
cell
as
a
failure.
Domaine
cells
can
be
anything
from
a
single,
a
single
rack.
That's
isolated
by
a
top-of-rack
switch
all
the
way
up
to
a
multi
data
center
region.
A
In
this
case,
I
have
the
cluster
set
up
as
two
cells
they're
set
up
with
each
one
of
them
is
spanning
three
regions
or
three
availability
zones
in
Google's
cloud.
So,
even
though
there's
three
different
failure,
domains
I've
set
it
up
so
that
the
cell
concept
in
Vitesse
is
all
three
of
them
to
accommodate.
For
that,
when
we
make
our
pods
and
I
will
go
ahead
and
show
you.
A
A
It
is
not
in
the
same
host
as
somebody
with
the
same
shard
and
that
it's
not
in
the
same
host
as
somebody
with
the
same
index
that
prevents
things
like
if
you
always
set
the
first
instance
of
a
shard
to
be
master
that
prevents
you
from
co-locating
your
master
on
hosts
in
this
case,
I've
set
it
to
hostname
because
again,
I
have
a
multi
cell
distribution.
So
if
one
of
my
availability
zones
go
down,
I
can
actually
failover
all
of
my
traffic
to
the
other
data
center
and
not
have
any
outages.
A
So
I
can
set
this
to
hostname
and
get
a
little
bit
more
flexibility
in
scheduling
this
takes.
You
know
full
kubernetes
annotation
support.
So
if
you
wanted
to
set
that
to
availability
zone,
you
can
go
ahead
and
do
that
and
it
will
go
low.
It
will
locate
your
pods
and
different
availability
zones
for
a
very
survivable
set
up.
That's
really
one
of
the
things
when
I
said
you
have
to
decide
how
much,
how
much
disruption
you
can
tolerate
as
you're
running
Vitesse,
running
things
in
the
same
AZ
leads
to
a
better
performing
cluster
info
test.
A
We
by
default,
run
with
something
called
semi
sink
semi
sink
for
my
sequel
means
your
transactions
won't
be
committed
until
at
least
one
replica
has
received
the
rights.
So
I
mean
think
about
that.
If
you
have
a
multi
multi
replicas
set
up
in
a
shard
that
replica
when
the
master
gets
a
right,
it
is
not
gonna
commit
it
until
at
least
one
of
its
replicas
has
received
it.
A
If
you
have
those
across
availability
zones,
that
means
every
single
right
transaction
that
you
have
is
going
to
incur
that
latency
across
availability
zones
before
it
will
acknowledge
that
can
substantially
decrease
performance,
but
it
does
mean
that
if
a
availability
zone
goes
down,
you're
guaranteed
not
to
lose
any
rights.
If
you
put
everything
in
the
same
availability
zone,
you
get
a
little
bit
more
performance
off
of
that
shard.
But
if
one
a
Z
goes
down,
then
that
shard,
the
data
at
that
shard
is
at
risk.
A
The
other
thing
about
data
centers
is
failing
over
a
data
center
is
actually
easier
than
just
failing
over
a
single
node
in
the
event
of
a
data
center
failure.
We
just
repaired
all
of
the
notes,
all
of
the
shards
away
from
that
data
center.
If
we're
stealing
with
a
single
node
outage,
then
we
have
to
worry
about
what
we're
gonna
do
with
the
individual
shards
that
may
or
may
not
be
located
on
that
node.
A
So
we
have
to
have
a
lot
more
operational
knowledge
and
again
it's
it's
easier
than
a
single
node
until
it
isn't
and
the
part
that
it
isn't
is
network
partitions.
So
in
all
of
this,
one
of
the
things
I
wanted
to
show
you
that
I
was
actually
just
working
on.
Couldn't
get
set
up
was
Orchestrator.
If
you
are
running
a
production
Vitesse
today,
you
should
run
Orchestrator
on
top
of
it.
Orchestrator
takes
everything
that
I
have
done
and
automates
it,
so
you
don't
have
to
go
through
any
command
line
utilities
or
any
web
interfaces.
A
Orchestrator
will
just
go
ahead
and
fix
your
my
sequel
deployment
when
it
breaks,
except
in
this
case
network
partitions,
tend
to
be
tough.
If
you
are
all
familiar
with
the
great
github
outage
of
2018,
it
ended
up
that
it
happened
because
of
a
15-minute
ish
Network
partition.
So
these
are
always
the
worst
kinds
of
failures
and,
unfortunately,
there's
not
much.
We
can
do
to
guarantee
that
nothing
bad
happens,
but
network
partitions
can
look
like
single
nodes
or
they
can
look
like
entire
data
centers.
A
So
when
we
are
setting
recovery
time
periods
on
Orchestrator,
you
have
to
take
into
account
the
fact
that
a
momentary
loss
of
connectivity
to
your
tablets
may
not
actually
be
a
failure.
It
may
in
fact
be
a
network
partition
and
it's
gonna
recover
itself.
The
network
is
in
fact,
not
reliable
and
you
have
to
deal
with
that.
B
Think
there's
one
one
obvious
question
which
people
may
have
is:
how
are
people
running
with
tests
in
production
and
how
they've
set
it
up?
There
are
a
few
few
different
ways
in
which
they've
configured
theirs
stitch
labs.
For
example,
they
rely
more
on
the
persistent
disk
of
kubernetes,
so
they
rely
on
the
disk
providing
the
necessary
resilience
so
which
means
that
when
a
node
goes
down,
they
actually
bring
back
the
part
under
that
and
type
act.
The
same
persistent
volume.
B
B
Slack,
for
example,
does
use
semi
sink,
but
they
use
it
with
a
twist.
What
they
do
is
they
have
multiple
AZ's,
but
any.
If
there
is
a
master
and
a
replica
on
the
same
availability
zone,
then
the
replica
within
that
availability
zone
does
not
give
you
a
semi
sink
AK,
which
means
that
when
you
commit
on
the
master,
the
data
has
to
at
least
be
received
by
a
replica
that
is
not
in
the
same
availability
zone.
So
that
is
another
common
way
by
which
people
have
configured,
and
there
are
people
who
just
play
it
safe.
B
They
rely
both
on
the
mounted
volume
and
the
semi
sink
because
they
just
want
the
other
these,
because
if
your
network
is
good,
the
cost
of
adding
a
semi
sink
replica
is
actually
not
much
so
might
as
well
pay
the
price.
The
trade-off
for
using
persistent
volume
versus
local
disk
is
local.
Disk
is
more
performant
because
there
is
absolutely
no
latency.
So
if
your
application
is
latency,
sensitive
people
prefer
to
do
the
local
disk
rout
and
then
not
worry
about
data
loss.
B
B
I'm,
not
a
bag,
a
very
big
fan
of
group,
replication,
I,
know,
I,
know
it
works,
but
I
don't
think
it
is
really
necessary
because
I
mean
there
are
so
many
deployments
that
have
worked
well
with
semi
sync
replication
that
and
it
is
so
much
simpler
and
easier
to
manage
that
I
know
it
works.
I
know
there
are
some
people
that
are
using
it,
but
I
haven't
felt
the
place
where
it
is
actually
needed.
B
B
The
writing
seems
to
be
on
the
wall
that
everybody
is
moving
to
kubernetes.
So
it's
just
a
question
of
how
long
people
are
going
to.
There
are
some
people
who
are
saying
that
it's
not
going
to
happen.
Some
people
are
saying
we
know
it's
eventually
going
to
happen
so
and
we
are
preparing
for
it.
But
the
good
news
is
that,
if
you
are
on
with
tests,
then
moving
to
kubernetes
is
pretty
easy.
It's
pretty
easy
to
move
into
that.
B
B
Oh,
that's
a
very
good
question,
so
the
question
is:
how
do
you
configure
sink,
pin
lock
and
in
ODB
the
like?
Basically,
how
consistent
do
you
want
your
my
sequel
to
be
so
the
what
we
did
at
YouTube
is
we
turned
off
I,
don't
know
what
the
exact
value
is
off.
We
basically
made
those
values
extremely
unsafe
because
by
making
those
values
extremely
unsafe,
you
get
a
huge
throughput.
B
A
B
B
B
B
Hubspot
uses
orchestrators
like
uses
Orchestrator
Oh
square
does
not
use
Orchestrator,
but
again
they
have
their
own
version
of
Orchestrator.
So
most
of
these
companies
that
have
adopted
youtube,
they
are
like
real
hardcore.
Let's
say
they
want
like
five
nines
of
availability
and
you
cannot
do
five
nines
of
availability
without
some
kind
of
automated
reparent.
Yes,.
B
B
B
The
the
the
semi
sink
configuration
is
actually
a
my
sequel
setting
and
you
acted
actually
have
to
set
it
at
the
database
level,
which
means
that
you
can
you
config,
you
configure
a
master
saying
that
I
require
at
least
one
AK,
and
you
configure
a
replica
saying
that
you
are
allowed
to
send
an
ACK.
So
that
is
how
so
it
is
global.
It
is
at
a
high
level.
Yes,.
B
B
There
is
a
brand
new
feature
that
we
are.
We
have
actually
got
a
proof
of
concept,
for
which
is
one
of
the
best
things
that
I'm
super
excited
about,
which
is
what,
if
you
have
an
M
M
by
n
relay
what,
if
you
have
a
many-to-many
relationship,
it
is
impossible
to
represent
that
in
a
shard
system.
So
what
we
are
doing
is
we
are
going
to
do
on-the-fly
replication
from
one
shard
one
shotting
scheme
into
another
sharding
scheme,
which
means
that
no
matter
which
way
you
join
the
join
will
always
be
a
local
joint.
B
So
I
have
present
I
presented
this
demo
in
queue
con
SF,
but
we
had
very
little
time
to
be
able
to
cover
that
here
and
most
people
are
interested
in
what
how
to
get
started
with
the
test.
But
this
feature
will
go
into
beta
in
about
two
months
and
it's
a
B.
It's
essentially
an
unsolved
problem
in
the
sharded
world,
because
the
data
I
can
only
be
in
one
place,
but
now
we
are
materializing
it
in
different
places
so
that
your
read
is
efficient.
A
So
I
was
been
aggressive
with
trying
to
get
a
high
availability
demo
to
show
off
in
45
minutes
and
cover
all
the
survivability
aspects,
but
it
will
go
up
on
our
github
on
the
Vitesse
community
github.
So
please
check
that
out.
Also,
if
you
are
interested
in
Vitesse
and
would
like
to
learn
more,
our
slack
community
is
the
best
place
to
reach
us.
You
can
go
to
Vitesse,
slack,
comm
and
sign
up.
A
We
would
be
glad
to
go
into
detail
about
any
questions
you
have
like
I
said:
look
on
github
I'm,
gonna,
put
all
of
the
files
that
I
have
for
the
cross
data
center
cross,
kubernetes
cluster
highly
survivable
cluster.
Up
on
there
and
I
may
even
record
a
video
of
me
actually
going
through
it.
It's
probably
gonna
take
a
few
hours,
but
it's
definitely
gonna
be
a
good
watch.
Also,
if
you
really
like
of
the
tests,
planet-scale
is
hiring
and
you
can
come
work
on
the
test
full-time.