►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Exploring High Availability in Kubernetes with Vitess - Jiten Vaidya, PlanetScale
As companies grow their infrastructure on the cloud in Kubernetes, questions of high availability arise. To be truly cloud native, you must be able to handle failure at any point in your stack. Vitess has many features that can help in failure identification and handling, such as vtgate-as-proxy and data recovery through duplication (instead of backup). Vitess also works with many third party tools like Prometheus for easy monitoring and observability. In this talk, PlanetScale cofounder and CEO Jiten Vaidya will present the case for planning for failure. Using a live demo, Jiten will show how designing for high availability using Vitess allows you to prepare for risk, avert disasters, and recover from catastrophic failure.
https://sched.co/Nrq4
A
It's
an
open-source
project
which
is
part
of
the
cloud
native
computing
foundation.
Witness
was
co-founded
by
suba
suba
Myron,
who
and
I
work
together
at
YouTube
back
in
2010
when
we
started
started
with
us
since
then,
witness
has
grown
quite
a
bit
and
witness
is
now
used
in
production
by
companies
like
slack
companies
like
Square
cash
I,
don't
know
how
many
of
you
actually
know
about
these
companies,
but
these
are
very
popular
companies
in
United
States
that
run
really
large
workloads,
Pinterest
YouTube.
A
All
of
these
companies
use
with
us
here
in
China,
JD
dot-com,
is
one
of
the
biggest
users
of
it
as
their
runs
run.
Tens
of
thousands
of
loads
on
the
tests
so
I'm
going
to
spend
some
time
talking
about
with
us
the
witness
architecture,
but
the
two
main
points
that
I
want
you
all
to
take
away
with
is
that
witness
is
used
for
horizontal
scaling
of
my
sequel
databases.
One
and
Vitesse
is
also
used
to
allow
you
to
run
stateful
were
closed
in
workloads
and
kubernetes.
A
As
you
all
know,
it's
easy
to
run
stateless
services
in
kubernetes,
it's
hard
to
run
stateful
workloads
in
kubernetes,
but
companies
like
HubSpot,
calm
and
Jerry.
Calm
have
been
using
the
tests
to
run
databases
and
kubernetes
over
the
last
two
or
three
years
with
us
is
born
to
run
in
kubernetes
because
with
us
ran
under
borg,
which
was
google's
orchestration
framework,
which
was
the
predecessor
to
kubernetes.
So
because
of
that
it
runs
really
well
in
kubernetes.
A
You
are
running
your
own
load
balancer,
but
we
ticket
onwards
is
all
Vitas
and
it
gives
your
application
the
view
that
is
talking
to
a
single
humongous
database
when
in
reality
it
could
be
talking
to
a
database
which
is
very
largely
horizontally,
shouted
with
hundreds
of
replicas
and
so
on,
and
not
just
one
database
like
this
multiple
databases
or
multiple
key
spaces
with
multiple
schemas
right.
Your
application
still
thinks
that
is
talking
to
one
humongous
database
right.
How
does
it
manage
that?
So?
A
The
very
first
component
is
VT
gate,
so
we
ticket
talks
my
sequel,
binary
protocol
to
app
servers.
It
just
basically
presents
itself
as
a
my
sequel.
Server
to
your
to
your
application
mitigate
also
has
a
full,
my
sequel,
parser
built
into
it.
It
parses
your
query,
looks
at
the
where
clause
and
decides
which
of
the
charge
the
query
needs
to
go
to.
Okay,
then
it
sends
the
query
you
over
G
RPC
to
the
VT
tablet
associated
with
the
mice
equality
to
which
it
wants
to
send
the
query.
A
Ok
does
not
send
it
directly
to
my
sequel
D.
It
sends
it
first
to
the
VT
tablet.
We
T
tablet
acts
as
a
guardian
or
a
protector
of
the
mice.
Equally,
every
time
there
was
a
YouTube
outage
because
of
any
reason
we
built-in
protections
that
would
keep
your
databases
alive
in
VT
tablet.
So
what
are
the
kind
of
kinds
of
things
that
we
T
tablet?
Does
we
T
tablet
does
connection
pooling?
We
T
tablet
gives
you
hot
row
protection.
It
C
realises
hot
rows
so
that
you
can.
A
A
So
it
also
puts
timeouts
on
your
queries
and
transactions
in
general.
Anything
that
a
consensus
DBA
would
do
to
keep
your
my
sequel
de
healthy
is
done
by
is
done
by
wiki
tablet
for
you
you,
so
that's
so
what
is
shown
on
the
right
hand.
Side
are
four
shards.
Each
child
has
one
master
and
multiple
replicas.
There
are
two
types
of
replicas,
the
replicas
and
big
data
replicas.
You
typically
run
your
batch
job
against
the
big
data
replicas
and
the
other
replicas
are
candidates
for
becoming
master.
A
A
One
of
the
coolest
things
about
Vitesse
is
that
it
does
not
make
decisions
about
how
data
is
charted
for
you.
It
allows
you
to
make
those
decisions
based
on
your
domain.
So
just
the
way
you
have
my
sequel
schema.
We
allow
you
to
have
what
is
called
V
schema
where
you
describe
to
Vitesse
how
you
want
your
reader
to
be
sharded.
A
So
what
you
do
is
that
for
every
table
you
tell
with
us
hey
chard
this
table
using
this
particular
column
and
depending
on
the
type
of
the
column,
we
allow
you
to
use
a
different
shouting
function
so
that
the
data
is
charted
uniformly
across
the
whole
key
range
right.
So
for
numeric
key
types,
it's
hash
or
wire,
binary
and
wire
Cal
types.
A
It's
it's
a
different
type,
so
there
are
six
different
predefined
shouting
functions
that
if
that
we
give
you,
but
not
only
that
we
also
give
you
the
ability
to
write
your
own
shouting
functions
and
because
of
this,
it's
very
easy
to
spin
up
clusters
which
are
jurisdiction.
Aware
for
GDP
are
compliant
clusters
using
using
vedettes
okay.
A
A
So
I
have
talked
through.
Most
of
these
points.
Here
is
how
witness
allows
you
to
run
well
in
kubernetes
right,
stateless
entry
point
to
allow
scaling
that
VT
Gate
is
a
stateless
proxy.
You
can
run
as
many
of
them
as
you
want.
Some
people
run
VT
gates
as
a
sidecar
to
their
application
server.
Some
people
just
run
a
farm
of
about
40
or
80
or
100
of
them
behind
the
load
balancers.
A
The
way
I
had
shown
it
in
the
in
the
application
in
the
diagram
before
you
can
litigate,
you
can
use
it
to
make
sure
that
only
your
read
traffic
is
only
load
balanced
across
local
replicas
in
local
cell.
You
can
even
have
a
group
of
VT
gates
to
make
sure
that
only
certain
key
spaces
are
visible
and
certain
are
not
so
VT
gate
is
a
very
versatile
proxy.
It's
stateless,
perfect
for
kubernetes,
so
that
so
that
your
applications
can
connect
to
it.
A
Vt
tablet
is
the
my
sequel
minder
that
I
talked
to
you
about
just
your
bicycle,
so
we
run
your
my
security
and
weekly
tablet
in
two
different
containers
in
the
same
pod,
and
it
just
works
really.
Well,
your
my
sequel
availability
is
really
high.
We
support
native
backup,
restore
health
checks
and
observability
so
that
kubernetes
services
etc
know
when
a
VT
gate
has
joined
the
quorum
and
when
the
vita
gate
has
gone
down.
All
of
that
is
supported
and
we
end
up
pushing
state
to
the
edges.
A
The
connection
strings
are
in
the
application
so
which
key
space.
If
you
want
real
after
right
consistency,
you
just
give
the
name
of
the
database.
So
if
the
name
of
your
database
is
user
DB,
if
you
give
the
name
of
the
user,
edit
databases,
user
DB,
you
get
rights
and
you
get
rid
of
the
right
consistency.
If
you
say
that
my
the
name
of
my
database
is
user,
DB
@
replica,
then
your
reads
get
then
your
reads:
get
load
balanced
across
the
replicants
okay.
A
So
all
of
that
you
get
and
now,
let's
jump
into
the
demo,
let
me
show
you
what
I'm
going
to
show
you
today,
which
is
that
I'm
we
are
going
to
run
a
global
stateful,
kubernetes
application.
I
get
I
showed
this
application
to
some
of
you.
You
all
can
go,
go
here
and
check
it
out.
It's
basically
an
application
for
rating
little
puppies
and
you
can
all
go
there.
34.8
e
dot,
120,
8.1,
30,
slash
poppers,
and
you
can
rate
these.
Please
make
sure
that
you
give
them
good
good
ratings,
they're
all
good
dogs,
okay.
A
So
this
is
the
application
that
we
are
going
to
talk
about
and
I'm
going
to
show
you
how
how
this
is
organized
in
with
us.
So
we
have
created
two
key
spaces:
one
called
doggers
and
one
called
called
the
cup.
The
doggers
key
space
has
four
shards
and
if
you
click
on
it,
you
can
see
the
these
four
shots.
Look
at
how
they
are
named.
So
the
key
space.
A
If
you
take
the
hexadecimal
key
space,
zero,
zero,
two:
four:
zero:
four
zero:
two:
eight
zero,
eight
zero
to
C,
0
and
C
0
to
FF
are
the
four
slices
of
the
key
space.
So
that's
how
we
named
charge.
If
you
click
on
each
char,
you
can
see
that
for
each
char
there
is
one
master
and
multiple
replicas.
Ok,
and
if
you
see
this
second
column,
which
is
the
cell,
you
can
see
that
the
master
is
in
u.s.
West
and
the
replicas
are
in
u.s..
Some
are
in
US,
east
and
some
are
in
asia
1.
A
So
each
shard
is
a
truly
global
cluster.
Ok,
now,
if
we,
what
I'm
going
to
do,
is
I'm
going
to
show
you
how
this
looks
like
if
I
connect
to
the
VT
gate
through
my
sequel,
client,
okay,
they
could
look
at
the
interesting
version
string
here.
We
kind
of
lie.
This
is
the
survey
server
version,
which
is
a
vertex.
Five
point,
five
point
ten.
They
say
this
is
you
know
this
is
a
blatant
hi.
It's
not
the!
We
are
running
five
point,
seven
underneath
and
you
can
give
whatever
connection
string
you
want.
A
So
it's!
These
are
scatter
gathers
there
that
are
being
sent
to
all
four
shards
that
counts
are
being
sent
back
to
you.
You
are
not
even
aware
that
you
are
talking
to
a
shout
that
database
just
wanted
to
show
you
that,
and
now
what
I'm
going
to
do
is
I'm
going
to
start
up
script
for
writing
into
this
database.
A
Okay,
so
this
is
going
to
keep
running
while
I'm
running
this
demo,
and
now
we
are
going
to
abuse
this
database.
I
am
going
to
first
delete
a
master
for
one
of
the
shards
and
see
what
happens
then
I'm
going
to
bring
a
whole
data
center
down
and
see
what
happens.
So.
The
idea
is
that
we
will
see
some
errors,
but
it
will
self
heal
without
me
having
to
do
anything
and
the
errors
will
go
away.
Okay,
that's
what
we
are
going
to
demonstrate.
A
So
the
first
thing
that
I'm
going
to
do
is
I
am
going
to
actually
first
before
I.
Do
that
I'm
going
to
show
you
a
plan
tree
parent,
so
there
are
times
when
you
know
that
you
need
to
bring
down
the
master.
How
do
you
achieve
that?
So
you
come
here,
and
you
say
what
I'm
going
to
do
is
instead
of
this
master
running,
is
your
uswest
I'm
going
to
choose
this
to
be
the
replica
which
I
want
to
make
the
master?
A
We
do
tablet
associated
with
the
new
master
waits
until
all
the
replication
logs
have
been
applied,
and
then
it
makes
that
the
master
resets
the
replication
for
the
whole
cluster
and
then
makes
the
entry
into
the
topo
server
in
which
all
the
information
about
the
cluster
topology
is
written
and
then
VT
gets
once
they
realize
that
a
new
master
is
there
all
the
queries
that
we
do
get
has
been
buffering.
They
start
sending
little
the
new
master,
the
point
being
that
your
app
never
sees
any
errors
right.
We
continue
to
write
these
into
it.
A
A
A
A
So
yeah
I
have
had
problems
connecting
to
the
kubernetes
cluster,
but
let's
see
so
what
I
am
what
I
was
trying
to
do
is
basically
I
was
going
to
so
while,
while
this
is
happening,
let
me
show
you
what
mechanism
we
use
for
making
sure
that
this
works
correctly.
So
it's
it's
another
open
source
tool
called
Orchestrator
and
that.
A
Shows
you
it
is
integrated
with
Vitas,
and
it
shows
you
the
cluster
topology
as
it
as
it
thinks
is.
It
is
so.
You
can
see
that
there
are
these
four
shots.
This
each
has
nine
instances,
and
if
you
look
at
the
topology,
you
can
see
that
it.
It
actually
shows
you
that
here
is
the
master
and
then
here
are
replicas
in
in
in
it
colors
them
differently,
because
they
are
in
different
cells
right.
So
what
I
am
and
I
am
trying
to
do?
A
A
A
Okay,
awesome,
so
the
part
we
are
getting
pods,
so
I'm,
gonna,
kill,
I'm
gonna
kill
one
of
the
masters
now,
okay,
so
the
V
T
tablet
demo,
seven
over
one.
That's
that's
the
master
that
we
saw.
We
we
had.
This
is
the
master
for
the
4080.
Okay,
so
I'm
gonna
kill
this
now
and
once
that
part
is
deleted.
We
should
start
seeing
errors
here.
C
A
A
C
A
A
A
So
these
three
are
dead,
so
it
cannot
get
to
them,
but
there
is
a
new
master
and
there
are
replicas
that
it's
it's
going.
It's
writing
too.
So
all
of
these
shards
have
reorganized
themselves
and
they
are
serving
traffic.
So
it's
a
basically
resilient
globular
distributed
cluster
that
is
being
mediated
by
the
tests.
B
B
C
A
When
the
shard
count
changes
it
basically
mostly
either
go
basically
with
as
supports
what
is
called
split,
shouting
or
merge
shouting.
Both
you
have
to
you,
have
to
provision
new
charts
and
then
Vitesse
does
the
job
of
rerouting
for
you,
while
continuing
to
serve
traffic
from
the
old
ones.
First,
it
does
a
stale
copy.
Then
it
does
what
we
call
split
replication
and
then
it
allows
you
to
catch
the
replication
catches
up.
It
allows
you
to
failover
from
all
master
to
new
master.
B
Yes,
that
is
the
beauty
of
the
tests
because
of
all
the
BT
gates
are
serving
all
the
traffic
you
can
choose
to
migrate,
just
the
read-only
traffic,
the
replicas
and
then
at
some
point.
Once
everything
is
done,
you
can
move
the
master
traffic
over.
That
makes
sense.
Other
questions.
Yes,
sorry!
Yes,.
E
A
A
When
I
removed
the
when
I
remove
replicas
when
I
removed
a
cluster
replicas
from
all
shards
which
were
in
that
cluster
got
removed
right,
so
there
were
always
some
replicas
which
had
all
the
data
which
were
part
of
the
serving
quorum.
So
that's
how
it
can
it
continues
to
so
I
said
us:
east
I
just
deleted
everything
in
u.s.
East,
but
there
are
replicas
in
u.s.
West
and
Asia.
One
were
still
part
of
the
cluster
and
we're
still
operating
correctly.
A
Contact
us
at
jetan
at
planet-scale
dot-com,
and
we
have
a
project
where
we
want
to
translate
all
of
it
as
documentation
into
Chinese
and
Rhoni.
Hey,
who
is
one
of
our
colleagues,
is
going
to
be
giving
the
talk
later
today
about
that
project
which
will
allow
you,
which
will
translate
all
of
it
as
documentation
into
Chinese.
D
A
Is
on
the
roadmap,
but
not
for
the
next,
at
least
not
for
the
next
year,
so
it's
not
that
hard
to
support
it
because
of
the
architecture,
but
what
we
need.
So
there
are
two
things
that
we
need
to
support.
One
is
the
protocol
that
a
Postgres
clients
talked
with
to
post
this
server.
That
should
be
pretty
straightforward.
Then
we
need
to
extend
our
parser
a
little
bit
so
that
there
are
certain
but
SQL
syntax
that
Postgres
supports,
but
my
sequel
doesn't
so
we
need
to
extend
our
parser
to
support
that.
A
A
So
there
is
about
eight
or
nine
month
worth
of
work
that
is
needed
to
do
that.
But,
right
now
our
hands
are
full
with
my
sequel,
but
we
will
probably
supported
within
a
year
to
year
and
a
half
as
our
company
planet-scale
grows.
But
in
the
meantime,
if
anybody
wants
to
help
it's
an
open
source
project,
thank.