►
From YouTube: One VTOrc To Rule Them All – High Availability In a Distributed Database System - Deepthi & Manan
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
One VTOrc To Rule Them All – High Availability In a Distributed Database System - Deepthi Sigireddi & Manan Gupta, PlanetScale
Speakers: Deepthi Sigireddi, Manan Gupta
Vitess is a scalable, highly available distributed database system built around MySQL. It achieves scalability through sharding and durability through replication. High availability is accomplished through a Vitess feature known as cluster management. The next generation cluster management service in Vitess is called VTOrc. Users can specify their durability rules as a system configuration, which is respected while performing planned failovers. VTOrc also performs failure detection with automatic failovers while honoring the durability rules. VTOrc is already running successfully in production in multiple deployments including at PlanetScale, and it will be Generally Available in Vitess release 15 (October 25). The session will provide an introduction to VTOrc and an outline of the theory that underpins its implementation, followed by a demo of its capabilities showing multiple failover scenarios.
A
A
A
A
Wittus
is
massively
scalable.
If
we
look
at
some
of
the
numbers,
the
largest
instances
of
wittest
run
to
20
000
terabytes,
which
is
20
petabytes,
22,
000,
replicas,
250,
000,
simultaneous
connections,
though
that
is
not
a
limit.
You
could
go
higher
and
so
on
and
one
of
the
ways
bitus
achieves.
The
massive
scale
is
through
sharding,
which
we
will
not
talk
about
in
this
talk.
But
we
have
a
maintainer
talk
tomorrow,
where
you
can
learn
more
about
how
we
test
does
sharding
wittus
is
also
highly
available.
A
A
In
addition
to
this
query,
serving
path
through
which
vtgate
receives
the
queries
and
routes
them
to
the
backend,
my
sequels,
there
is
also
a
control
plane,
and
this
is
where
VTR
comes
in.
There
is
a
witness
control
Daemon,
which
takes
management
commands
from
users
through
CLI
or
the
web
UI
and
executes
them.
There
is
VT
admin
which
Powers
our
web
UI
and
then
there's
VT
Arc.
A
So
let's
go
through
a
little
overview
of
VTR
and
then
we
will
have
a
demo.
What
is
the
problem
that
VTR
is
trying
to
solve?
The
main
problem
that
VTR
is
trying
to
solve
is
resiliency
to
my
SQL
failures,
so
we
are
running
MySQL
in
a
replicated
mode,
but
even
then,
if
the
primary
MySQL
goes
down,
we
have
to
deal
with
that
failure
mode
and
we
need
to
be
highly
available
and
that's
the
problem
that
PTR
is
solving.
However,
while
ensuring
High
availability.
A
A
A
The
integration
worked
well
enough
most
of
the
time,
but
not
all
of
the
time.
In
addition
to
that,
the
way
durability
was
provided
by
Vitas
was
through
semi-synchronous
replication,
which
we
will
go
through
in
a
minute
and
that
that
configuration
in
invitus
didn't
really
work.
Well.
All
of
the
time
we
discovered
failure
modes
where
the
semi-synchronous
mode
setting
would
be
lost,
whatever
the
user
intended
would
not
get
set
on
the
MySQL.
A
A
Let's
get
a
little
deeper
into
what
VTR
is
doing.
We
talked
about
what
functionality
it
provides,
but
how
does
it
do
that?
So
VTR
is
the
agent
that
is
monitoring
the
bitters
cluster
and
detecting
the
failure.
So
it
has
logic
to
check
for
certain
conditions
at
periodic
intervals
and
flag
failures,
and
once
it
detects
a
failure,
it
goes
through
a
repair
or
recovery
mechanism.
A
The
way
VTR
achieves
durability
is
through
replication
or
in
general.
The
way
wittus
achieves
data
durability
is
through
replication,
so
one
primary
more
than
one
replica
gives
us
guarantees
that
any
transaction
that's
accepted
by
a
primary
is
replicated
to
at
least
one
replica,
so
that
in
case
of
a
disk
failure
or
a
node
failure,
you
don't
lose
the
data
that
a
client
thinks
has
been
persisted
and
the
way
we
achieve
High
availability
is
through
failover.
A
So
when
a
failure
is
detected,
we
call
it
an
unplanned
leader
election,
but
in
the
normal
course
of
things
you
have
to
upgrade
software.
You
have
to
upgrade
Hardware
whatever
it
is
in
the
normal
course
of
things.
You
want
to
be
able
to
do
planned
failovers
so
that
the
system
remains
available
through
maintenance
operations
and
that's
Witnesses.
General
strategy
for
high
availability
and
VTR
is
the
agent
that
enables
those
things.
A
Performance
over
theoretical
correctness,
not
to
say
that
the
system
is
not
going
to
be
correct,
but
just
that
we
did
not
prioritize
writing
a
formal
specification
and
validating
it.
We've
taken
an
engineering
approach
to
solving
the
consensus
problem
and
we
are
not
interested
in
Purity
in
the
sense
that
we
don't
have
to
be
the
consensus
algorithm.
So
witness
has
a
topology
server
which
can
be
hcd
or
zookeeper
or
console
where
we
store
some
State
and
it
is
used
for
Discovery.
A
We
have
also
separated
the
planned
leader
election
from
the
unplanned
leader
election.
So
during
the
there
is
the
normal
operation
of
the
system,
which
is
very
high
QPS
we
will.
We
expect
witness
systems
to
get
millions
of
queries
per
second
for
reads
or
rights
versus
leader.
Elections
are
much
rarer,
so
planned
leader
elections
in
a
kubernetes
environment,
maybe
once
a
day
or
once
a
week
or
in
a
non-kubernetes
environment,
maybe
even
rarer
and
unplanned
leader
elections
are
expected
to
be
relatively
rare.
A
A
So
in
a
planned
leader,
election
revocation
is
actually
much
simpler
than
in
an
unplanned
leader
election,
and
this
is
one
of
the
reasons
we've
separated
those
two
in
a
planned
leader
election,
because
the
current
leader
is
still
available.
You
just
ask
them
to
step
down
before
selecting
a
new
leader,
so
it's
an
orderly
transition.
A
A
new
leader
is
chosen.
It's
possible
that
there
are
still
some
completed
requests
that
have
not
been
propagated
to
all
of
the
replicas.
The
system
will
make
sure
that
they
get
propagated
to
all
of
the
replicas.
So
that's
what
a
planned
leader
election
looks
like.
So
as
an
example,
the
leader
changed
and
the
yellow
arrows,
which
represent
replication
have
all
moved
over
to
the
new
leader.
A
A
A
Mysql
has
a
synchronous,
replication
mode
and
a
semi-synchronous
replication
mode
and
with
the
right
parameters,
it's
you
can
configure
semi-synchronous
replication
in
such
a
way
that
any
right
to
the
primary
will
be
blocked
unless
an
acknowledgment
is
received
from
at
least
one
replica,
and
the
acknowledgment
from
the
replica
doesn't
mean
that
that
transaction
has
been
completely
applied
to
the
replica.
It
only
means
that
the
replica
has
received
it
and
persisted
it
in
a
way
on
on
disk.
Essentially
so
what
the
durability
policy
is
defining
is
basically
how
this
is
configured
for
with
us.
A
A
Most
people
use
one,
but
there
may
be
situations
where
people
want
to
use
two
acknowledgments,
which
replicas
are
eligible
to
send
those
acknowledgments
and
so
on,
and
what
durability
policies
give
operators
of
witness
is
increased
flexibility
in
their
configurations
in
their
topology
in
how
they
lay
out
their
replicas
for
worldwide
replication
or
across
data
centers
regions
availability
zones.
Those
kinds
of
things
so
now
I'll
hand
it
off
to
Manon.
B
For
that
introduction,
and
now
that
we've
looked
at
what
durability
policies
are
and
how
a
test
tries
to
ensure
durability,
we
can
go
ahead
and
look
at
the
revocation
phase
in
a
little
more
detail
and
look
at
exactly
what
m
is
that
deeply
alluded
to
in
a
few
slides
ago?
So
how
do
we
know
that
we
have
been
able
to
reach
sufficient
tablets
to
guarantee
safety?
To
answer
that
question,
we
first
have
to
look
at
a
different
concept
called
intersecting
quorums.
B
This
set
of
tablets,
which
is
capable
of
accepting
rights,
is
one
Quorum
for
accepting
rights.
There
can
be
many
such
possible
quorums
in
order
to
guarantee
safety
during
the
revocation
phase,
we
should
be
able
to
reach
at
least
one
tablet
from
each
of
the
possible
quorums.
The
intent
is
that
if
there
are
unreachable
tablets
which
can
form
a
quorum,
they
would
accept
rights
which
we
don't
want
them
to
do.
B
B
Let's
say
that
we
were
in
a
scenario
where
we
had
six
tablets,
in
which
you
only
require
three
tablets
to
accept
transactions.
If,
for
example,
something
was
Network
partitioned
like
so
that
you
had
two
VTR
instances
running
but
on
two
different
sides
of
the
network
partition
with
three
tablets
on
either
side
in
this
situation,
if
you
do
not
check
for
the
revocation
phase
at
all,
each
of
those
VTR
instances
can
actually
promote
one
primary
and
that
primary
will
be
able
to
accept
rights,
because
you
only
require
two
more
semi-sync
Acts.
B
So
we
would
have
one
primary
elected
here,
one
primary
elective
here,
and
that
would
not
be
a
correct
situation,
because
you
could
very
well
be
in
a
split
brain
scenario.
Two
of
these
primaries
accepting
rights.
They
can
diverge
because
they
could
have
accepted
different
rights.
They
aren't
replicating
to
each
other
because
they're
Network
partitioned,
so
in
this
case
it
would
not
be
safe
for
any
of
the.
We
talk
instances
to
proceed
because
from
this
we
talk
instances
perspective.
B
B
If,
for
example,
on
the
other
hand,
we
were
Network
partitions
such
that
five
tablets
lie
on
the
same
side,
and
one
tablet
was
on
the
other
in
this
case
that
we
talk
instance
can
make
forward
progress
because
it
can
reach
five
tablets,
and
it
is
enough
to
guarantee
revocation
because
the
one
tablet
which
is
there
on
the
other
side,
it
can't
actually
accept
rights,
because
two
semising
acts
are
required
and
we
can
reach
enough
tablets
to
guarantee
that
that
tablet
never
gets
that
many
acts.
I
hope
this
concept
is
a
clear.
B
Let's
take
a
look
at
the
demo,
so
the
demo
that
we're
going
to
look
at
it's
actually
going
to
be
in
two
parts.
The
first
part
is
going
to
use
a
semi
sync
durability
for
that
we're
going
to
use
a
semi-syncy
space
and
the
durability
policy
that
we're
going
to
use
is
semi-sync.
The
way
that
it
works
is
that
any
replica
can
be
the
primary
tablet.
B
B
B
So
if
you
take
a
look
at
the
quorums
that
can
accept
transactions
because
any
tablet
can
be
the
primary
and
any
other
tablet
can
send
a
semi
sync
act.
Essentially,
any
two
tablets
together
can
accept
the
right,
because
one
of
them
can
be
the
primary
and
the
other
can
send
the
act.
For
example,
in
this
case,
100
can
be
the
primary
and
101
can
send
the
act,
and
that
is
enough
to
accept
the
right.
So
overall,
this
this
is
what
the
Quorum
said
for
accepting
rights
would
look
like
any
two
tablets
can
accept
a
right.
B
B
But
if
you're
able
to
reach
three
tablets,
100
100
to
103.
That
is
enough,
because
there's
only
one
tablet,
which
is
unreachable,
the
101
tablet-
and
it
can't
accept
rights
on
its
own
if
you're
able
to
reach
all
the
tablets.
That
is
enough.
Two
and
one
is
not
so.
In
our
example,
we
have
four
tablets,
one
of
which
is
the
primary.
B
There,
in
that
much
time,
VTR
promotes
a
different
tablet,
which
is
Zone
102
as
the
primary
and
the
previous
one
get
demotes
itself
to
a
replica
which
is
no
longer
in
a
serving
state.
But
in
this
case,
if
we
go
ahead
and
delete
one
more
primary
failure,
let's
say
in
this
case
our
cluster
had
another
primary
failure.
This
time,
VTR
won't
be
able
to
make
any
progress.
The
reason
is
that
there
are
two
failures
now
and
these
two
tablets
had
they
be
Network
partition.
They
could
have
accepted
rights,
so
VTR
can't
reach
them.
B
It's
not
going
to
be
able
to
make
forward
progress,
which
is
what
the
flexibility
is.
What
PTR
provides
because
you're
able
to
configure
the
amount
amount
of
semi-sync
ax
you
require?
It
is
up
to
you
to
Define
how
much
of
a
durability
you
want,
if
you
send,
if
you
set
the
number
of
semising
Acts
requirement
to
something
higher,
let's
say
five
semi
saying
X
and
you
have
10
tablets
running.
B
The
next
example
that
we're
going
to
look
at
is
a
crosstalk
durability
policy.
This
policy
is
similar
to
the
semising
policy,
apart
from
one's
one
difference.
So
the
way
that
we
Define
a
cell
is
a
failure
domain.
It
could
be
a
Amazon
region.
It
could
even
be
your
confidence
in
different
providers.
You
could
say
that
AWS
is
one
zone
and
gcp
is
the
other
Zone.
B
However,
you
decide
to
do
that
so
once
we
have
a
cell,
the
Crosshair
durability
policy
all
is
different
in
the
semising
one
is
that
the
replica
acts
have
to
come
from
a
tablet
which
is
in
a
different
cell.
So
essentially,
when
a
primary
accepts
a
right,
the
semi
sync
Act
is
coming
from
a
different
cell,
so
you
are
guaranteed
that
your
right
is
persisted
in
two
cells.
B
B
The
tablet
in
zone
one
can
send
some
icing
acts
to
all
the
other
three
and
can
receive
semi-sing
acts
from
all
the
other
three,
because
it
lives
in
a
different
Zone,
but
not
the
other,
because
Zone
one
or
two
hundred
and
Zone
2201
both
are
in
the
same
cell,
which
is
zone
two.
So
they
can't
send
some
icing
acts
to
each
other,
which
is
what
the
white
line
signifies
there.
B
If
you
take
a
look
for
the
quorums
of
verification,
reaching
200
and
201
is
not
enough,
because
you
have
tablet
202,
which
can
accept
rights
with
300.
300
202
201.
That
is
enough,
because
there's
only
one
replica,
that's
left,
300
and
201
is
also
enough,
because
if
you
reach
300
you're
able
to
guarantee
that
none
of
them
are
actually
able
to
make
forward
progress
because
300
is
part
of
all
of
those
Quorum
sets.
B
B
B
Okay,
so
now
Zone
2
200
is
the
primary.
If
we
go
ahead
and
take
this
primary
down
as
well,
this
time
too,
VTR
will
be
able
to
fix
this
failure,
and
the
reasoning
is
again
in
the
durability
policies,
because
300
and
201
are
enough
200
and
202.
Even
though
there
are
two
tablets,
they
can't
send
some
icing
acts
to
each
other,
so
they
can't
accept
rights
between
themselves,
and
this
is
where
the
difference
between
the
semi-sync
policy
and
the
CrossFit
policy
comes
in,
because
the
failures
are
in
the
same
cell.
B
Cross-Sell
policy
can
make
further
progress,
even
though
semi-sync
could
not.
So
in
the
case
of
semising
policy.
Two
failures
was
enough
that
we
talk
was
not
able
to
make
forward
progress
in
this
case
it
can.
It
has
promoted
someone
else
as
the
primary
Zone
to
201
and
the
things
are.
Things
are
resumed,
but
any
more
failures
than
this
even
201
tablet
going
down.
That
would
again
result
in
a
scenario
where
VTR
can't
make
progress,
because
it
won't
be
able
to
make
forward
progress.
B
There's
only
one
tablet
available
and
you
need
one
semising
act
to
even
get
it
to
even
accept
any
rights,
and
that
concludes
the
demo
for
both
of
those
key
spaces.
I
would
like
to
talk
a
little
bit
more
about
the
custom,
durability
policies,
so
the
way
that
the
durability
policy
works
is
that
you
only
need
to
answer
three
questions
when
you
implement
your
own
custom
durability
policy.
This
is
encoded
as
an
interface.
So
first
thing
you
need
to
answer
is
who
all
can
be
the
primary?
B
B
The
second
question
is
how
many
semi-sync
acts
does
each
primary
require,
and
this
can
be
customized
for
each
tablet.
That
can
be
the
primary.
You
could
say
that
you
have
too
much
confidence
in
one
of
the
primaries.
Let's
say
primary
tablet
a
it
requires
threesome
icing
acts,
but
you
you
want
tablet
B,
to
only
require
one
semi
sync
act.
That
is
also
possible,
and
the
final
question
is
the
number
of
surmising
Acts
that
are
defined,
who
all
tablets
can
send
these
acts
based
on
who
the
primary
is.
B
With
all
of
these
three
questions,
you
can
actually
Implement
your
own
custom
durability
policy,
based
on
the
constraints
that
you
have
and
the
trade-offs
that
you're
willing
to
make,
apart
from
the
primary
failures,
BTR
also
handles
many
other
failure,
scenarios
that
MySQL
instances
generally
run
into
so
it
maintains
the
cluster's
desired
state
if
the
primary
goes
into
a
read-only
state
or
the
replicas
replication
is
stopped,
and
so
on.
The
list
of
failures
are
listed
here
and
many
more.
These
are
all
the
things
that
we
talk
is
able
to
handle.
B
So
if
you
have
VTR
running
it's
kind
of
an
automated
scenario
where
you
just
you,
don't
need
to
look
at
anything
until
something
drastic
happens,
something
that
you
did
not
prepare
for
or
you're
not
expecting
according
to
your
durability
policies
to
handle.
If
that
happens,
then
manual
intervention
is
required.
Otherwise,
we
talk
is
an
automated
process
that
will
take
take
care
of
your
cluster
and
keep
it
healthy
all
the
time.
B
Thank
you
and
we're
going
to
move
on
to
the
question
and
answer
round
before
we
do
that
there
are
some
resources
that
we
have
tomorrow.
There
is
another
bit
test
talk
which
is
going
to
talk
about
sharding
and
new
features
in
vitess
at
11
55
and
the
blog
post
series
is
linked
here
and
the
video
documentation.
Please
leave
feedback,
you
can
use
the
QR
code
for
that.
We
would
love
to
hear
from
you
as
to
how
the
talk
was
and
how
we
could
improve.
Thank
you
any
questions.
A
A
C
And
the
second
question
is
I
noticed
you
were
careful
not
to
say
m
stands
for
majority
of
the
I
I
noticed
that
you
said
like
M,
the
parameter
M
doesn't
stand
for
majority.
You
know
for
tablets
right,
but
for
scenarios
where
you
know
m
is
let's
say
only
two
replicas
and
we
can
have
a
potentially
a
split
brain
scenario
like
when
it's
less
than
majority.
How
can
one
you
know
recover
from
those
scenarios
or
how?
How
can
we
reconcile
University
scenarios
where
split
brain
can
happen?
When
m
is
less
than
majority.
A
B
Yes,
so
the
way
that
m
is
written
is
that
it
can
actually
even
be
variable,
based
on
the
failures
that
you
have
in
the
cross
sale
policy.
If
you
actually
have
a
failure
of
the
Zone
one
zone,
300
tablet
the
one
which
is
in
the
other
Zone
the
only
tablet
there.
That
is
enough,
because
you
only
have
one
tablet
running
in
that
zone.
In
that
case,
none
of
the
other
tablets
can
make
progress.
B
So
if
we
go
back
to
that
slide
in
this
case,
if
this
tablet
goes
down,
there's
actually
enough
to
bring
down
the
entire
cluster,
because
you
have
only
one
tablet
in
a
different
Zone,
but
that
also
means
that
you
can
handle
multiple
failures
from
this
zone.
Two
tablets
going
down
from
here
is
what
we
saw
in
the
example.
That's,
the
m
is
even
it's
not
even
a
constant
based
on
the
durability
policy.
B
It
could
be
one,
it
could
be
two,
it's
all
dependent
on
the
kind
of
failures
you
have
and
the
end
of
the
question
that
you
had
as
part
of
the
second
question
was
what
happens
and
how
you
reconcile.
If
you
end
up
in
a
split
brain
scenario,
the
way
that
VTR
works
is
that
it
will
not
allow
you
to
get
into
a
split
drain
scenario.
B
It
will
stop
rights
at
that
point,
so,
for
example,
in
the
situation
that
I
described
over
here
that,
if
you
were
Network
partition
like
so-
and
you
had
one
of
the
primaries
running
in
this
cluster
bit
went
down
in
this
case.
Vtr
won't
promote
anything
else,
because
it's
not
safe
to
proceed.
It'll
actually
wait
for
manual
intervention
to
come
in,
in
which
case
the
operator
can
decide
how
he
wants
to
proceed.
In
this
scenario,
he
can
actually
take
down
all
of
these
three
tablets
and
just
promote
one
from
here.
B
A
Let
me
answer
that
question,
so
the
basic
principle
that
is
preventing
that
from
happening
is
Quorum
intersection.
So
what
we
are
saying
is
that
if
you
have,
let's
say
five
tablets
all
together
and
you
require
two
acknowledgments
for
for
the
primary
to
accept
a
right,
then
when
you
try
to
revoke
the
leadership
from
the
current
primary,
you
need
to
form
a
quorum
that
includes
at
least
one
of
those
replicas
that
is
able
to
acknowledge.
A
So
in
that
example,
of
five
tablets
with
two
acts,
you
need
at
least
three
replicas
other
than
the
primary
to
agree
that
this
current
primary
is
going
to
step
down.
So
the
Quorum
for
normal
operation
is
different
from
the
Quorum
for
leadership
election
and
they
have
to
intersect
and
that's
the
same
Principle,
as
has
been
published
in
the
literature,
Quorum
intersection.
C
A
So
the
question
was
about
what
happens
if
the
kubernetes
API
server
is
unavailable,
so
VT
Arc
needs
to
be
able
to
communicate
with
the
topology
server
that
we,
where
we
store
state
that
doesn't
necessarily
depend
on
the
kubernetes
API
server.
So
as
long
as
the
networking
in
the
kubernetes
cluster
is
functional
and
the
topology
server
is
reachable,
we
will
still
be
able
to
perform
an
election.