►
Description
Building a Cloud-Native SQL Database [I] - Alex Robinson, Cockroach Labs
While compute management solutions like Docker and Kubernetes have rapidly advanced in recent years due to the proliferation of cloud infrastructure, storage options have been left behind in comparison. Conventional SQL databases have tacked on clustering options that help a little bit, but what would it take to build a strongly-consistent database specifically for cloud-native deployments.
This talk takes a deep-dive into the design of one such database, CockroachDB, which is being built from the ground up via open source. We will look specifically at how to achieve the easy deployment and management of a scalable, self-healing, strongly-consistent database with techniques such as dynamic sharding and rebalancing, consensus protocols, lock-free transactions, and more. We will then explore how you can both contribute to it and use it to build scalable, resilient applications that can be deployed to any cloud infrastructure.
About Alex Robinson
Alex Robinson is a software engineer at Cockroach Labs, the startup leading the development of the fully open source (Apache-licensed) CockroachDB project. Previously, he was a senior software engineer at Google, where he spent his last two years as a core developer of both Kubernetes and Container Engine. Prior to that, he worked on Google's cloud logging and monitoring infrastructure.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So,
if
you
think
of
traditional
sequel
databases
going
back
to
post
grads
or
my
sequel
or
all
the
proprietary
solutions,
they
typically
run
on
a
single
machine.
You
might
shard
them
or
you
might
set
up
master-slave
replication,
but
they're
typically
built
from
the
ground
up
to
run
on
a
single
machine
and
that
single
machine
becomes
a
single
point
of
failure
and
it
also
limits
your
ability
to
scale
up.
As
you
want
to
store
more
data
or
process
more
queries.
You
have
to
buy
bigger
and
bigger
machines,
rather
than
just
adding
more
machines.
A
That's
a
real
limitation
from
an
operational
perspective,
so
a
cloud
native
database
is
one
that
can
survive
failures,
scale
out
horizontally
and
is
really
easy
for
the
operator
to
manage,
and
this
describes
a
number
of
existing
databases.
It
just
turns
out
that
they're
all
no
sequel
systems,
so
you
think
to
all
the
no
sequel,
databases
you
know
and
love
most
of
them
have
these
horizontal
scalability
and
failure,
recovery
properties
to
at
least
some
extent
they
can
replicate
data
they
can
spread
across
datacenters
for
some
of
them,
but
they're
all
lacking
a
couple
really
important
features.
A
The
sequel
databases
have
traditionally
offered
to
developers
and
that's
going
back
to
consistency
in
transactions.
So
when
you're
using
a
sequel
database
developers
get
this
nice
benefit
that
they
can
trust
the
data
in
it
is
all
up
to
date,
there's
nothing
stale.
No,
you
know
replication
is
clobbering
data.
You
can
trust
that
transactions
won't
interact
with
each
other
in
bad
ways,
so
that
everything
you
write
and
read
from
the
database
is
consistent
and
safe,
and
this
makes
writing
systems
significantly
easier.
A
If
you
can
rely
on
these
properties,
if
you've
ever
built
a
system
that
uses
a
no
sequel
database,
you
probably
spent
some
time
up
front
having
to
around
this.
If
you
can
think
back
to
your
experience,
you
might
have
ran
into
bugs,
while
you
were
operating
it
due
to
stale
data
or
due
to
your
application
crashing
in
the
middle
of
the
multi-step
operation.
These
things
cost
a
lot
of
developer
time,
a
lot
of
developer
effort.
A
It
also
caused
bugs
for
customers,
so
Google
created
a
system
called
spanner
that
many
of
you
have
probably
heard
of
that
was
the
inspiration
for
cockroach
TV
and
spanner
in
the
paper
justifying
it.
The
authors
of
the
paper
wrote
that
we'd
much
rather
have
developers
occasionally
looking
into
latency
problems
caused
by
using
transactions
too.
Much
then
to
have
those
developers
have
to
work
around
the
lack
of
transactions
all
the
time,
it's
much
better
for
Google
as
a
business.
A
If
their
developers
don't
have
to
worry
about
the
lack
of
transactions
and
if
they
can
just
trust
that
the
database
works
the
way
they
expect.
So
they
went
to
this
huge
expense
of
both
building
and
operating.
The
system
and
found
it
to
be
more
than
worth
it
for
everyone
to
start
using
it
and
I
think
the
same
thing
is
true
of
a
lot
of
systems
out
there
outside
of
Google.
A
And
you
know
it
would
be
great
if
more
systems
provided
consistency
and
transactions.
But
when
no
sequel
databases
came
around,
the
fundamental
motivation
for
building
them
was
availability
and
scalability.
You
had
these
large
web
scale.
Companies
trying
to
run
massive
distributed
services
that
couldn't
work
with
just
a
single
machine
database
backing
them.
They
needed
to
be
able
to
stay
up
all
the
time
to
scale
out
across
multiple
machines
and
to
survive
all
sorts
of
failures,
because
these
systems
need
to
be
up
all
the
time
or
else
the
companies
are
losing
money.
A
And
so
it's
not
the
people
developing.
You
know
big
table
or
Cassandra
or
back
back
when
they
did
we're,
throwing
out
consistency
in
transactions
because
they
didn't
think
they
were
valuable
orcs.
They
thought
they
were
too
confusing.
It
was
purely
a
matter
of
prioritization
that
it
was
too
hard
to
package
them
into
the
database
and
they
thought,
availability
and
scalability
were
more
important,
which
has
absolutely
been
true.
You
know
a
lot
of
great
systems
have
been
built
on
top
of
these
databases
that
don't
have
consistency
or
transactions.
A
Today,
you
have
to
make
this
choice:
you're,
picking
between
multiple
options,
you're
making
a
trade-off,
no
you're,
either
picking
a
sequel
database
that
is
great
for
your
developers,
but
probably
tough
on
your
operations
team
who
has
to
deal
with
this
lack
of
horizontal
scalability
or
you're
picking
a
no
sequel
database,
which
is,
you
know,
easier
to
operate
in
in
some
ways.
But
it's
a
lot
tougher
on
your
developers
who
have
to
make
this
extra
effort
to
build
around
the
databases,
issues
and
that's
where
cockroach
DB
is
trying
to
come
in.
A
That's
why
we're
building
this
new
system?
We
want
a
database,
that's
scalable
that
can
survive
failures.
That
is
strongly
consistent,
and
that
gives
the
standard
sequel
interface
that
everyone
is
familiar
with,
and
it's
all
being
built
out
in
the
open
and
open
source
out
on
github,
and
we
think
that
this
is
something
that
will
make
building
applications
easier
for
everyone,
while
still
making
the
database
easy
to
operate
for
your
reliability
teams
and
so
I'm
going
to
talk
most
of
today
about
how
the
system
works.
A
So
how
you
can
build
a
system
that
replicates
data
that
scales
horizontally.
The
survives
failures,
while
still
providing
consistency
without
having
to
sacrifice
performance
and
so
I'm,
going
to
focus
on
three
main
areas:
kind
of
starting
from
the
core
of
the
system
and
in
terms
of
how
data
gets
spread
around
a
cluster
of
machines,
then
getting
into
how
we
keep
that
data
consistent
amongst
them
and
how
we
provide
arbitrary
transactions.
On
top
of
that
and
before
we
get
into
too
much
detail,
it's
worth
understanding
how
the
system
is
laid
out.
A
A
It's
you
know
it's
still
a
beta
product
right
now
we're
hoping
to
go
window
in
the
next
month
or
so
yes,
so
the
system
is
split
up
into
a
bunch
of
layers.
You
know
where
the
transactional
key,
V
key
value
store,
can
rely
on
this
data
being
distributed
across
a
bunch
of
machines
and
can
rely
on
that
data
being
replicated
consistently
and
can
rely
on.
A
You
know,
let's
thought,
a
solid
layer
that
rep
that
persists
the
data
onto
disk,
so
we're
gonna
start
by
looking
at
this
middle
layer
that
tracks
where
the
data
is
amongst
all
the
machines
in
the
cluster
and
keeps
track
of
it,
so
that,
when
a
query
comes
in,
we
know
exactly
where
the
data
that
it
wants
is
located
and
when
you're
working
with
a
distributed
system.
That
is
managing
data.
The
first
question
you
should
typically
ask
is:
how
is
that
day,
to
split
up
onto
the
machines?
A
And
how
do
you
find
any
given
data
that
you
want,
because
that's
very
fundamental
to
how
everything
else
within
the
system
works,
and
so,
when
you're
asking
how
its
split
up
and
how
its
located?
There
are
two
approaches
that
been
used
in
a
lot
of
previous
systems,
kind
of
as
prior
art.
The
first
is
hashing,
which
is
seen
in
systems
like
Cassandra,
where
to
find
where
a
given
piece
of
data
is
located.
A
You
hash
the
key
using
some
hash
function,
that's
predetermined
and
then
use
that
resulting
value
to
index
into
the
machines
in
the
cluster,
and
this
is
great
because
you
know
exactly
where
any
given
key
is
located.
Just
based
on
the
value
of
the
key.
You
don't
have
any
sort
of
separate
lookup
structure.
A
So
you
pick
arbitrary
dividing
lines
that
create
reasonably
trunks
for
your
for
your
data
and
we
call
each
of
those
chunks
a
range
because
it's
a
range
of
the
key
space,
and
so
once
you
have
that
you
don't
know
where
any
of
those
chunks
live,
because
given
a
key,
you
just
know
that
it's
in
some
chunk
somewhere.
But
that
doesn't
tell
you
where
it
is.
So
you
have
to
add
this
range
indexing
structure
on
top,
which
is
a
little
more
work
than
you
need
in
a
hash
space
system.
A
So
this
range
index
simply
keeps
track
of
all
the
different
ranges
that
are
in
the
system
and
what
machines
they're
currently
located
on
and
given
that
you
can
do
a
lookup
pretty
easily
by
checking
the
index
to
find
which
range.
The
key
you
want
to
find
is
in
getting
the
locations
of
that
range
within
the
cluster
and
then
going
to
ask
one
of
those
locations
for
the
key.
A
Another
nice
little
perk
of
splitting
up
your
data
in
this
ordered
way
is
that
when
you
get
too
much
data
in
a
range,
you
can
just
split
it
in
half.
You
know
pick
some
point
in
the
middle
break
it
into
two
new
ranges
and
update
your
indexing
structure
in
some
consistent
manner
in
a
hash
based
approach.
If
you
end
up
with
too
much
data
on
one
machine,
you
have
to
either
add
a
bunch
more
machines
so
that
the
data
falls
elsewhere
or
change.
A
Your
hashing
function
in
some
way,
which
can
be
much
trickier
operation
than
just
splitting
a
range
in
half,
and
so
once
you've
split
this
range
in
half
they
can
each
be
managed
separately,
moved
around
to
different
machines
independently
and,
of
course,
we
keep
multiple
copies
of
you
changed.
Yes,.
A
So
all
of
this
is
within
the
system.
It's
it's!
It's
invisible
to
the
user.
Data
can
be
split
up
automatically
by
the
system
so
that
they're
in
reasonably
sized
chunks,
the
user
can
also
request
to
have
us.
Pre
split
ranges
it
for
performance
reasons,
if
they'd
like,
but
in
general,
we'll
split
things
when
the
data
gets
too
large
to
be
in
a
single
range
for
performance
reasons
and
availability
reasons,
and
the
index
gets
automatically
updated
all
the
time
when
this
happens
and
it's
all
invisible
to
the
user.
Does
that
answer
your
question?
A
So
we
need
to
keep
multiple
copies
of
everything,
because
this
is
a
distributed
system.
Machines
can
fail.
We
need
to
plan
for
that.
So
by
default
we
keep
three
copies
of
every
range
you
can
configure
it
to
be
more
if
you'd
like,
and
we
rather
than
having
some
external
control
process.
That
manages
where
all
the
data
is
located.
It's
built
into
the
nodes
themselves,
so
each
node
is
gossiping
information
about
itself
so
that
you
know
node
one
has
some
idea
of
how
much
data
is
on
node
four.
How
many
ranges
that
are
there?
A
A
Node
4
will
generally
accept
it
and
take
it
and
then
node
3
can
get
rid
of
it,
which
lightens
the
load
on
node
3
and
makes
it
that
if
a
node
fails,
there
will
be
less
recovery,
that's
needed,
and
similarly
this
will
continue
happening
until
the
replica
thinks
or
the
system
thinks
that
it's
mostly
balanced,
so
ekend
range
will
get
replicated
over
there.
And
at
this
point
the
cluster
is
mostly
balanced
and
the
same
thing
happens
if
a
node
fails
just
a
little
bit
faster
and
more
urgently.
A
So,
for
all
this
we
make
the
query.
The
range
is
about
64
megabytes
in
size,
which
is
just
an
experimentally,
determined
value
that
performs
pretty
well,
it's
kind
of
a
fight
between
making
them
smaller,
which
makes
them
easier
to
replicate
around
the
cluster
and
cheaper
to
do
so
and
cheaper
to
split
into
multiple
hats.
A
If
the
caches
need
to
be
bigger,
it
slows
down
the
system
a
little,
and
so
this
method
of
splitting
up
data
into
ranges
and
distributing
them
around
a
pool
of
nodes
is,
you
know,
has
been
in
a
bunch
of
a
bunch
of
pre-existing
systems,
most
notably
such
things
like
HBase
or
the
Google
databases,
and
that's
how
we
keep
data
spread
around
the
cluster.
Are
there
any
questions
on
that
before
I
move
on
to
how
we
keep
data
in
sync?
A
Know
so
there
are
a
couple,
so
the
this
indexing
structure
is
actually
kept
as
a
range
in
the
cluster.
So
it's
located
on
three
five
seven,
however
many
nodes
and
then
when
a
note
like
another
node,
receives
a
request
to
do
a
lookup.
It
has
to
check
with
the
index
to
find
where
that
is,
but
as
a
basic
optimization
keeps
a
cache
of
the
the
ranges
that
it
looked
up
recently.
A
So
once
you
have
a
bunch
of
data
in
your
cluster
as
soon
as
you
have
multiple
copies
of
it,
the
immediate
question
is:
what
happens
if
they
get
out
of
sync,
or
how
do
you
keep
them
from
getting
out
of
sync,
because
when
you
only
have
one
copy,
it's
really
easy.
You
know
exactly
what
the
truth
is.
You
have
multiple
copies,
they
can
get
out
of
sync,
one
of
them
can
get
corrupted,
as
was
mentioned,
and
so
you
have
these
multiple
copies,
but
you
have
to
keep
them
in
sync.
A
A
traditional
solution
is
to
use
a
primary
secondary
replication.
This
is
what
you
see.
In
most
traditional
social
databases,
some
more
modern
systems
have
started
using
what's
called
consensus.
This
is
what's
used
in
@cd,
which
backs
kubernetes,
and
primary
secondary
application
is
a
very
simple
idea
on
the
surface.
The
idea
is
that
you
have
a
primary
note
at
all
times.
It
receives
all
the
writes.
It
writes
its
incoming
requests
to
disk
and
sends
a
belong
to
some
secondary,
which
also
writes
them.
A
A
You're
increasing
your
latency
by
adding
a
round-trip
to
every
write
and
it's
again
very
tricky
on
failover,
and
it's
really
hard
to
get
this
right
in
practice,
and
so
consensus
tries
to
make
this
simpler
in
practice
by
making
it
harder
to
understand
up
front
yeah,
it's
kind
of
flipping
the
tables
a
little
bit.
You
know
there's
a
whole
bunch
of
consensus.
Research,
that's
been
done
there,
you
see
systems
like
Peck,
so
it's
likely
you
stamp
replication
like
graft.
A
As
you
see
there,
and
it
looks
a
little
bit
like
this
in
raft
a
right
comes
into
the
leader,
a
leader
is
usually
elected
as
a
performance
optimization
so
that
all
rights
go
through
the
leader
and
you
don't
have
to
deal
with
multiple
things
proposing
and
conflicting
data.
So
at
all
times
a
leader
has
been
elected
by
the
quorum.
A
If
the
leader
gets
a
right,
it
replicates
it
onto
a
follower
as
soon
as
any
of
the
followers
have
written
that
right
to
disk,
the
data
is
considered
committed,
even
before
anything's
been
acknowledged
as
soon
as
it's
on
disk
the
data
is
committed,
and
then,
in
the
usual
case
the
acknowledgement
will
come
back
and
the
leader
can
acknowledge
the
client
notice
that
this
runs
as
fast
as
the
fastest
quorum
of
nodes.
If
you
have
a
couple
slow
nodes
on
the
tail,
it's
not
a
big.
You
can
have
two
nodes.
A
You
know
running
together
in
Germany,
another
node
in
Australia,
and
it
doesn't
matter
if
that
third
node
is
really
far
away.
Your
commits
can
proceed
as
fast
as
your
two
replicas
that
are
closest
together,
so
you
can
get
really
good
performance
on
all
your
reads
and
writes
in
Germany
and
just
leave
Australia
catch
up.
In
the
background.
A
The
interesting
part
is
what
happens
when
things
go
wrong,
and
so,
if
you
have
the
same
setup
and
a
write
comes
in,
but
the
leader
crashes
after
it
writes
its
disk,
then
what
will
happen
is
that
one
of
the
other
nodes
will
be
elected
as
a
leader,
and
so
the
second
node
here
got
elected
as
a
leader
and
it
never
got
the
key
cherry.
So
the
state
of
truth
in
this
cluster
right
now
is
that
cherry
is
not
part
of
these
systems,
because
only
one
of
the
three
nodes
has
it.
A
So
if
a
read
comes
in
for
that
key,
the
second
node,
the
new
leader
will
be
able
to
say
that
that
key
does
not
exist
and
they
know
that
has
it
will
be
corrected
in
the
background.
So
the
the
read
that
comes
in
for
cherry
will
return
that
it
does
not
exist
in
the
system
and,
as
I
mentioned
we
use
raft,
we
share
the
implementation
with
that
CD,
so
they
started
it
out.
We've
been
contributing
to
it
along
the
way
comments.
You
know
really
really
well
hardened
implementation.
A
We
run
one
consensus
group
for
each
range
of
data,
so
in
that
past
example
with
the
three
and
four
nut
read
for
nodes,
where
we
had
three
ranges.
Each
of
those
ranges
is
a
separate
consensus
group
and
in
a
real
cluster
since
they're,
each
only
64
megabytes,
you
can
easily
get
thousands
tens
of
thousands
hundreds
of
thousands
of
these
ranges,
which
is
a
lot
of
consensus
groups
to
be
managing
and
that's
where
we
differ
from
at
CD
quite
a
bit,
because
their
whole
system
is
one
consensus
group.
A
So
we
have
to
have
a
lot
bunch
of
engineering
to
work
around
this
to
avoid
overloading
the
network
with
all
sorts
of
raft.
Background
traffic,
but
Raph
provides
an
incredibly
solid
foundation
for
the
system.
Even
though
there
are
some
engineering
complications
involved
in
using
it,
we
can
truly
rely
on
any
right
to
RF
group
being
atomic
and
durable.
It's
either
committed
or
it
isn't,
and
once
it's
committed,
it
sits
pretty
reliably
there.
So
this
is
a
great
foundation
for
the
system.
A
A
We
default
to
the
serializable
isolation
level,
which
is
the
this
first
isolation
level
that
provides
true
isolation
that
doesn't
have
any
interactions
between
between
queries.
This
is
actually
stronger
than
some
single,
no
databases
default
to
it,
which
is
kind
of
funny,
and
in
order
to
do
this,
we
can't
just
copy
what
single
node
databases
do
we
have
different
constraints.
You
know
we
can't
rely
on
a
single
disk.
Write
to
you
know
a
time
a
click
emit
something
because
we
have
multiple
disks
potentially
spread
across
the
world.
A
Some
of
the
things
that
we
do
to
work
around
this
are
to
rely
on
an
a
raft
group
right
being
somewhat
analogous
to
a
single
disk
block
right.
So
when
a
transaction
comes
in,
we
create
a
record
for
it.
That
record
has
some
ID
for
the
transaction
has
its
state
some
other
metadata,
and
when
we
do
rights
within
the
transaction,
we
tagged
the
right
the
written
data
with
that
ID
to
indicate
that
it
might
not
be
committed
yet.
A
But
if
you
want
to
check
whether
it's
committed
go,
look
up
that
transaction
ID
see
whether
or
not
it's
done
and
then
when
the
transaction
finishes.
We
just
mark
that
single
transaction
record
which
lives
within
a
raft
group
as
finished,
and
so
that
raft
group
right
is
atomic,
and
so
that
makes
the
transaction
commit
atomic
and
as
for
locking,
we
have
to
use
an
optimistic
concurrency
control
protocol
instead,
which
effectively
amounts
to
detecting
conflicts
as
they
occur.
A
Rather
than
using
locks
to
prevent
them
from
occurring,
so
we
have
to
detect
them
and
then
abort,
one
of
the
transactions
or
retry
one
of
the
transactions
as
necessary,
and
so
I'm
gonna
go
through
a
couple
small
examples
here,
just
to
give
an
idea
of
what
it
looks
like
in
the
simple
case
where
everything
goes
well,
a
transaction
comes
in.
We
create
a
record
for
it
somewhere,
usually
on
the
same
range
as
the
first
right
for
efficiency
reasons.
A
And
so
again,
that's
that's.
What
happens
when
things
go
well,
the
interesting
part
of
the
transaction
protocol
is
what
happens
when
there
are
conflicts
and
when
things
go
wrong.
So
in
a
read,
conflict,
a
transaction
starts
and
writes
a
key
when
other
transaction
comes
in
and
tries
to
read
that
key,
and
it
sees
that
there's
this
transaction
ID
still
attached
to
that
written
key,
which
means
that
it
might
not
be
committed.
A
What
what's
interesting
here
is
that
there
are
some
time
stamps
involved
here.
So
if
transaction
2
is
timestamp
is
less
than
transaction
ones,
it
can't
just
go
ahead
or
it
can
just
go
ahead
and
read
the
value.
But
if
action
twos
timestamp
is
greater
than
transaction,
one
says
initially
and
might
have
to
push
the
timestamp
back
in
time,
because
it'd
be
illegal
to
return
that
the
key
doesn't
exist
if
a
transaction
later
comes
in
and
commits
that
key
at
an
earlier
time
stamp.
A
So
with
distributed
transactions,
the
timestamps
can
get
a
little
fuzzy,
but
we're
able
to
to
manage
the
timestamps
as
necessary
to
ensure
correctness
and
a
right
conflict,
and
we
have
two
transaction
records
both
try
to
write
the
same
key.
So
in
this
case
one
of
them
is
going
to
have
to
be
aborted.
A
Both
transactions
can't
write
the
same
key
at
the
same
time.
You
know
in
a
single
node
database.
The
second
transaction
would
just
get
blocked
on
the
lock
just
like
in
the
previous
read
example,
the
read
would
just
get
blocked
on
transaction
ones
lock,
but
since
we
don't
have
locks,
one
of
these
transactions
has
to
be
aborted
in
order
to
to
complete
successfully
so
based
on
some
randomly
generated
priorities.
You
know,
transaction
two
will
decide
whether
or
not
to
abort
itself
or
to
abort
the
existing
right.
A
A
You
know
these
real
valuable
guarantees
on
top,
and
so
real
quick
just
want
to
mention
how
to
deploy
it.
If
you
want
to
try
it
out
in
the
interest
of
being
cloud
native,
it's
a
single
statically
compiled
binary,
written
and
go.
The
UI
is
packaged
in
so
you
can
just
hit
a
different
port
on
the
binary
to
get
an
admin
UI.
That
gives
you
all
sorts
of
insight
into
what's
running
in
the
cluster.
A
A
It
doesn't
matter
at
all
and
it's
just
gonna
balance
and
heal,
no
matter
which
one
you
get
rid
of,
and
so
yeah
starting
a
cluster
is
super
easy
now,
just
run
cockroach
start
on
a
machine
if
you
want
to
join
more
machines
in
the
cluster,
just
start
them
up
with
the
address
of
any
of
the
nodes
in
the
cluster
and
they'll
connect
gossip,
their
information
and
join
the
join
the
the
consensus
groups.
Now
we
have
a
docker
image,
we
have
a
kubernetes
configuration.
We
have
a
helmet
art
if
you're
into
home.
A
A
A
A
Yes,
so
for
correctness
reads:
do
have
to
go
to
the
current
leader
and
actually
it's
even
a
little
stronger
than
that
they
have
to
go
to
what
we
call
a
current
lease
holder,
which
is
a
mechanism
built
on
top
of
raft,
because
otherwise,
without
this
extra
safety
that
we
add
in
even
sending
all
reads
to
the
leader
would
be
sometimes
incorrect
unless
those
reads
also
go
through
raft,
which
is
an
issue.
That's
come
up
with
the
number
of
systems,
including
STD
so
reads,
do
have
to
go
to
the
leader.
A
Yeah
yeah,
so
Kyle
worked
with
us
last
fall
and
you're
going
to
hold
a
whole
bunch
of
tests
against
us.
He
found
two
bugs
that
he
was
able
to
generate
inconsistencies
with
after
you
know,
tons
and
tons
of
pounding,
and
they
were
both.
You
know,
logic
errors
that
that
better
unit
testing
could
have
caught,
and
so
after
those
two
bugs
were
fixed,
he
wasn't
able
to
find
any
true
faults.
A
A
So
it
is
a
CP
system,
we're
not
claiming
that
in
a
during
a
partition,
all
nodes
will
be
able
to
make
forward
progress.
If
you
have
a
partition,
the
larger
side
of
the
partition
should
be
able
to
make
forward
progress,
but
anything
on
a
smaller
side
is
unlikely
to
be
able
to
do
so.
So
it's
fully
a
CP
system,
but
by
you
know,
replicating
data.
We
make
it
so
that
individual
things
going
down
doesn't
doesn't
hurt.
Availability.
A
Yeah,
so
you
know,
things
like
graft
have
made
this
much
easier
than
they
would
have
been
five
years
ago,
because
previously
you'd
have
to
try
to
implement
Paxos,
which
is
a
very
tough
task.
But
now
you
can
find
a
good
implementation
of
Raft
in
all
sorts
of
languages
and
build
a
system
based
on
that
I
mean
there's
a
ton
of
engineering
involved,
but
it
can
work
as
a
as
a
system.
If
you
put
in
enough
time
and
optimize
it
right.
A
Yes,
so
that's
something
that
we've
been
working
on
a
lot
over
the
last
six
months
or
so
it's
what
we're
calling
distributed,
sequel
where
we
basically
take
sequel,
queries
that
are
touching
a
lot
of
data
and
turn
them
into
something
akin
to
a
MapReduce,
where
we
farm
out
different
parts
of
the
query
to
different
machines
and
able
it
to
all
happen
in
parallel.
It's
still
slower
than
a
single
machine,
of
course,
because
we
are,
you
know,
moving
large
amounts
of
data
across
the
cluster
and
we
offer
a
feature
called
interleaved
tables
such
that.
A
If
one
table
has
a
foreign
key
constraint
on
another
table,
you
can
actually
interleave
it
underneath
the
table
that
it
has
the
foreign
key
constraint
on
so
that
joins
on
that
foreign
key
are
very,
very
cheap,
because
the
data
is
all
co-located
within
the
same
range,
so
that
can
help
with
certain
use
cases.
But
in
the
general
case
yes,
joins
can
be,
can
be
slow
even
with
the
distributed.
A
A
It
gets
information
about
the
cluster
via
that
gossip,
Network
and,
and
once
it's
in
that
gossip
network,
other
nodes
will
start
giving
it
data
splitting
arrange
to
create
a
new
range
or
creating
a
new
range
from
scratch
has
to
go
through
the
raft,
consensus
protocols
to
ensure
safety
and
so
Etsy
d.
Similarly
has
logic.
You
know
in
the
raft
implementation
for
adding
new
members
removing
members
in
a
safe
way.
A
Yeah,
so
it
scales
linearly.
We've
done
testing
on
that
up
to
64
notes,
we're
adding
each
new
node
as
a
linear
amount
of
additional
write
and
read.
Throughput
are
using
for
a
ballute
numbers
depend
a
lot
on
what
your
workload
is
and
on
what
kinds
of
machines
you
have,
but
we've
hit
a
hundred
thousand
QPS
on
a
recent
ycs
b
test
and
it
does
scale
and
yearly
as
long
as
you're,
not
adding
more
and
more
contention
as
you
scale
up
in
the
back.
A
A
Yes,
so
connecting
through
H
a
proxy
or
something
like
that
is
recommended,
so
that
you
can,
you
can
connect
to
any
node
in
the
cluster
and
query
for
any
data
and
that
works.
But
if
you
are
connected
to
only
a
single
node,
then
that
client
stops
working
as
soon
as
that
single
node
is
not
available
here.
A
A
It
will
wait
for
a
short
period
of
time
before
aborting
and
this
is
drastically
improved
performance
on
highly
highly
contended
workloads,
and
that
was
one
of
the
issues
in
the
Jepsen
testing
was
that
a
fear
was
was
hammering
single
rows
with
all
of
his
rights,
and
so
they
were
all
waiting
on
each
other,
I
mean
being
aborted,
but
adding
adding
the
short
waiting
period
drastically
improved
the
amount
of
retries
and
throughput.
That
was
able
to
be
done.
A
A
A
The
object,
relational
mapping,
libraries
that
are
in
a
bunch
of
languages
and
some
of
them
use
some
very
obscure
edges
of
postgrads,
and
so
we've
been
over
time,
adding
more
and
more
of
those
edges,
but
we
certainly
haven't
hit
all
of
them
yet
so
we
expect
most
applications
to
work,
but
it's
very
possible
you'll
hit
a
feature
that
we
have
not
yet
implemented
and
we
hope
you'll
tell
us
and
we
implement
it.
Usually
it's
not
that
big
of
a
deal
to
throw
them
in.
A
A
A
The
only
enterprises
would
need
are
going
under
a
separate
license,
so
there's
still
open
source
but
they're
under
what
we
call
the
cockroach
community
license,
and
so
these
features
that
are
planned
right
now
are
distributed
back
up
in
your
store,
rather
than
single
node,
like
a
slightly
slower
back
up
in
restore
that's
widely
available
and
then
per
row
geographical
placement
of
your
data.
So
right
now
you
can
configure
where
your
data
lives
on
a
table
level.
So
you
say
this
table
lives
in
these
data
centers
or
in
these
countries
or
whatnot
at
the
enterprise
level.
A
A
A
So
we
rely
much
less
than
spanner
in
the
sense
that
we
use
a
different
commit
protocol,
so
spanner
spanners.
Commit
protocol
requires
very,
very
low
latency,
like
single-digit
millisecond
latency,
for
commits
to
perform.
Well,
we
allow
up
to
hundreds
of
milliseconds
even
a
second
of
clock,
offset
between
nodes,
depending
on
your
environment,
what
with
still
with
good
performance.
So,
but
if
your
clock
offset
between
nodes
exceeds
the
pre-configured
amounts,
we
will
kill
one
or
more
of
those
nodes
processes
to
avoid
potentially
inconsistent
reads
that
can
happen.