►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How To Build a Distributed System (And Should You?) - Rebecca Bilbro & Patrick Deziel, Rotational Labs
In this talk, we’ll tell the story of how we built our very own eventually consistent system which is currently deployed in production clusters across the US, Germany, and Singapore -- including all the mistakes we made along the way. We’ll walk through how we leveraged tools like gRPC, Kubernetes, LevelDB, and Prometheus to implement two new open source projects that serve as the heart of our system. We’ll also confess all the ways we messed up during the process — from struggling to debug protocol buffer errors, to tangling up send and receive goroutines, to reasoning about the phases of replication. It won’t all be pretty, but we hope you’ll benefit from the lessons we learned, including the most important lesson — that you *can* build your own distributed system. We’ll close out by talking about why rolling our own system (in spite of all the headaches and mistakes) made sense for our use case, and why it might also make sense for you. Attendees will walk away with a hearty introduction to distributed systems concepts, as well as a to-do list of things they can investigate in their own systems to determine how they might be able to reduce concurrency-related bugs and/or consistency-related costs, improve maintenance, and reach more daily active users around the world.
A
Welcome
to
kubecon
Welcome
to
our
talk
on
how
to
build
a
distributed
system,
and
also
should
you
we
are
very
excited
to
be
here.
This
is
my
first
kubecon
so
I'm,
just
as
excited
to
learn
as
I
am
to
sort
of
tell
you
about
what
we've
been
working
on
so
who
are
we?
We
are
Patrick
desil
and
Rebecca
bilbro.
A
We
both
work
for
a
company
called
rotational
labs
and
rotational
Labs
builds
small
to
medium-sized
geodistributed
systems,
so
these
are
not
kind
of
like
Netflix,
Facebook
Twitter
scale
distributed
systems
but
they're
distributed
throughout
the
world,
and
and
because
of
that,
we
encounter
a
lot
of
interesting
problems
in
this
space
and
we're
going
to
talk
about
some
of
those
with
you
today.
There's
a
lot
of
overlap
between
the
problems
that
we
encounter
and
probably
a
lot
of
the
solutions
that
you
are
working
on
or
thinking
about
in
your
work.
A
So
we're
also
hoping
that
you'll
come
up
and
chat
with
us
later.
Hopefully,
you'll
be
able
to
recognize
us
and
you'll
come
find
us
and
to
tell
us
about
what
you
do
and
what
kinds
of
systems
that
you're
working
on
so
I
would
be
remiss
not
to
mention
our
team.
So
the
rotational
Labs
team
is
also
geo-distributed,
so
we're
from
all
over
the
world
and
I
wanted
to
give
a
special
shout
out
to
Karen
and
Benjamin.
A
Who
did
a
lot
of
the
experimentation
that
kind
of
led
to
us
coming
up
with
this
talk,
so
the
the
geoping
project
that
we'll
talk
about
is
something
that
they
kind
of
incubated
and
so
I
want
to
give
them
a
special
shout
out
and
if
you're
curious
about
what
we
and
the
rest
of
the
team
are
working
on.
I
would
encourage
you
to
check
out
rotational.app
and
you
can
get
a
little
sneak
peek.
A
A
This
is
hard,
if
you
know
anything
about
crypto
right,
blockchain
doesn't
really
support
the
sharing
of
the
kinds
of
information
that
you
would
need
in
order
to
do
auditing
as
a
government
official
right,
there's
like
like
you
know,
there's
no
way
to
look
up
somebody
right.
If
they're
it's
a
wallet
address,
it's
not
a
person
right,
so
it's
it's
definitely
a
little
bit
of
a
challenge,
and
so
this
is
kind
of
what
we
needed.
A
We
were
tasked
with
with
working
on
this
for
a
non-profit
group
of
people
who
are
trying
to
work
in
this
space,
and
so
this
is
what
we
built.
It's
kind
of
you
can
picture
it
as
like.
A
global
Yellow,
Pages
kind
of
that
allows
you
know
different
virtual
asset
service
providers
to
look
each
other
up
and
to
exchange
secure
private
information.
That
would
allow
governments
to
do
audits
if
they
need
to,
and
so
I
want
to
emphasize
that
this
is
secure
information.
This
is
private
information
that
is
being
exchanged
right.
A
A
So
the
very
first
problem
that
we
had
to
solve
is:
where
do
we
store
the
public
certificates
right?
So
you
need
to
be
able
to
look
up
the
public
certificate
for
your
counterparty.
We
got
to
store
that
data
somewhere.
Where
should
we
store
it?
Should
we
store
it
in
Singapore,
because
Singapore
is
the
place
that
sort
of
was
the
fastest
moving
in
terms
of
government
regulation?
So
it's
maybe
it
makes
sense
to
store
the
data
there.
But
if
we
store
the
data
there,
then
anybody
who's
doing
transactions,
not
in
Singapore.
A
Not
you
know,
co-located
in
with
the
Singapore
is
going
to
pay
an
extra
latency
fee
which
that
doesn't
really
sound
fair.
It's
not
really
in
the
spirit
of
a
distributed
Ledger
right,
so
we
need
to
figure
out
some
way
to
distribute
the
system,
so
that
brings
us
to
what
is
a
distributed
system
so
when
I
say
distributed
system.
Hopefully
some
of
you
already
kind
of
have
this
picture
in
your
mind.
You
know
of
a
group
of
servers
that
are
sort
of
arrayed
together
in
some
way.
A
A
The
way
that
this
works
is
that
each
peer
in
the
system
has
a
copy
of
the
database,
so
they
can
respond
independently
to
requests
that
come
in
concurrently
right.
So
why
do
they
even
need
to
work
together
if
they
all
have
a
copy
and
how
do
they
even
work
together?
Well
kind
of
a
a
side.
Note
you
know
tangent
here
is
that
they're
going
to
work
together
using
grpc
grpc
is
Google's
flavor
of
remote
procedure
calls,
and
so,
if
they
weigh
of
you
know,
you
can
define
an
API
with
protophiles.
A
You
can
compile
that
into
the
language
of
your
choice,
and
then
you
can
run
that
code
on
two
different
machines
and
because
both
machines
recognize
the
RPC,
they
can
actually
execute
commands
on
a
remote
machine,
and
that's
that's
how
this
is
going
to
work.
Why
do
they
need
to
cooperate?
Is
because
you
know
when
users
are
making
changes
to
Independent
peers.
Those
peers
at
some
point
need
to
synchronize
so
that
they
can
share
data.
A
That's
been
added
by
different
users
around
the
world,
and
sometimes
we
are
going
to
get
into
this
situation
where
two
users
have
concurrently
made
a
change
change
to
the
same
object,
but
with
different
values,
and
then
we
have
to
figure
out
how
to
decide
who
wins,
and
that
is
one
of
the
big
problems
in
distributed
systems.
There's
actually
a
lot
of
ways
to
solve
that
problem
and
it
kind
of
depends
on
what
your
business
case
is
and
what
you
need.
A
So
if
you
need
strong
consistency,
if
you
need
every
peer
to
always
have
the
same
answer
forever,
then
you
need
strong
consistency,
but
then
you're
going
to
pay
a
cost.
You
know
it's
going
to
take
more
time.
It's
probably
going
to
be
more
expensive
right.
If
you
you
know,
if
you
can
relax
those
requirements
a
little
bit
and
do
something
more
eventual,
then
you
know
you
might
have
a
problem
you
might
be.
You
might
be
able
to
have
lower
latency
and
higher
availability,
but
sometimes
you'll
see
a
stale.
A
You
might
get
a
stale
read
so
there's
kind
of
trade-offs
here.
For
us
we
picked
eventual
consistency.
It
was
a
good
budget
friendly
Choice.
Basically
it
you
know,
it
guarantees
that
if
everybody
stops
making
updates
eventually
everybody
all
the
peers
will
agree
on
what
the
data
should
be
whoops.
So
I
want
to
kind
of
shout
out
also
some
of
the
things
that
we
built
our
Solution
on
top
of
this
is
an
open
source
solution.
We
built
it
for
a
non-profit,
so
it's
all
out
kind
of
you
can
look
at
the
code.
A
We
built
two
main
packages:
honu
and
turtle
they're
both
names
for
Turtles,
so
there's
kind
of
a
Turtle
theme
going
on
right
here.
So
we
built
these
two
open
source
packages.
They
are
built
on
other
open
source
packages
like
level
DB
and
Prometheus
and
protocol
buffers.
But
the
way
to
kind
of
think
about
it
is
that
honu's
job
is
to
add
metadata
versioning
on
top
of
level
DB,
so
that
we
can
compare
objects
and
decide
who
should
win.
A
And
then
turtle
is
in
charge
of
actually
coordinating
synchronizations
between
peers,
and
so
you
know
basically
appears,
will
periodically
synchronize
they'll,
compare
their
version
vectors
and
then
make
updates
according
to
a
bilateral
anti-entropy
algorithm
which
I
won't
go
into.
But
you
can
definitely
ask
me
about
it
after
and
I
will
go
into
great
detail,
so
I'm
going
to
turn
it
over
to
Patrick
to
talk
a
little
bit
about
some
of
the
things
that
we
learned
along
the
way.
B
Great
thanks
Rebecca,
so
the
first
thing
I
want
to
talk
about
is
usually
in
kubernetes
we're
kind
of
trained
to
think
about
pods
in
terms
of
like
we
don't
want
to
get
too
attached
like
so
there's
a
kettle,
not
not
pets,
mentality,
which
means
that
we're
viewing
these
pods
as
really
vehicles
of
execution,
not
necessarily
any
form
of
a
state.
They
don't
really
stick
around
too
long.
B
We're
expecting
kubernetes
to
be
able
to
destroy
them
and
create
them
a
will,
but
this
is
kind
of
contrary
to
what
record
was
just
talking
about,
because
what
we
really
want.
Is
we
want
that
metadata
right
in
in
the
when
we're
creating
those
queries
to
the
databases?
So,
in
order
to
actually
perform
this
replication,
we
need
pods
to
be
addressable,
which
is
kind
of
easier
said
than
done
on
kubernetes.
B
B
So
what
this
means
in
terms
of
kubernetes
is
that
we
can
accomplish
this
using
a
kind
of
a
combination
of
a
few
different
things.
We
can
use
a
stateful
set
in
kubernetes,
which
is
basically
a
way
to
start
a
bunch
of
pods
which
have
an
Associated
identity
and
also
enforce
the
idea
of
uniqueness
in
pods,
so
we're
never
starting
the
same
pod,
the
same
identity
over
again
when
we
have
to
replay
things
or
change
configuration.
So
that's
my
aspect
to
it.
B
B
So
there's
another
problem
which
we
encountered,
which
has
to
do
more
with
how
the
client
interacts
with
our
system
so
we're
thinking
about
building
Geo
distributed
systems.
It's
really
important
to
consider
where
our
users
are
actually
coming
from
because
they're
coming
from
all
over
the
world,
so
we
don't
want
to
have
them,
give
them
like
undo
latency
when
they
don't
need
to
right.
B
So,
if
I'm
connecting
from
France
I,
don't
want
to
go
all
the
way
over
to
the
Singapore
server
in
order
to
make
a
request,
but
there's
a
bit
problem
with
using
something
like
Google
Cloud
load
balancer,
because
we
want
our
clients
to
be
able
to
connect
over
grpc
for
performance
reasons,
and
this
is
not
something
that's
very
easy
to
do
with
a
Google
load
down
service.
So
that's
one
issue.
The
other
issue
is
TLS
termination
when
we
get
to
load
balancer.
B
So
what
that
means
is
that
if
you
don't
know
what
it
is,
it's
basically
the
point
at
which
we
interrupt
traffic
and
we
decrypt
it
into
unencrypted
traffic
and
sometimes
like.
This
is
a
something
we
very
much
need
to
do.
If
you
want
to
look
at
what's
actually
coming
into
the
private
Network
or
things
like
that.
B
For
our
case,
it's
actually
something
that
we
don't
want
right,
because
we're
enforcing
mtls
throughout
the
entire
all
the
way
from
the
clients
to
the
grpc
endpoint
Services,
where
our
our
Turtle
might
be
or
another
another
web-facing
service
might
be.
So
we
really
need
to
enforce
grpc
or
mtls
across
the
entire
connection.
So
one
thing
that
we
can
do
is
we
can
push
the
routing
up
to
the
DNS
layer,
so
this
is
a
way
of
getting
around
that
problem,
right
of
not
being
able
to
use
the
grbc
with
a
pre-mensioned
load
balancer.
B
What
happens
is
that
it
goes
directly
to
the
grpc
or
service
because
they
already
have
the
IP
address
so
this
come.
This
is
a
coming
to
the
project
we
built
called
geoping,
which
is
basically
in
short
terms.
It's
just
a
grp
service.
That
knows
where
it
is,
and
it's
just
a
glorified
version
of
what
the
maybe
the
bash
or
Linux
ping
is
right.
B
So,
instead
of
like
pinging
and
getting
IP
address,
you
ping
and
you
get
a
region
where
the
closest
server
to
you
is
the
reason
why
we
built
this
is
that
we
wanted
a
way
to
actually
test
that
our
DNS
routing
is
working
since
it's
not
really
something
that
we're
doing
we're
relying
on
another
external
service.
For
that.
So
what
happens?
Is
that
it's
a
pretty
simple
service
if
you've
never
looked
at
a
grpc
surface
definition.
This
is
like
probably
the
most
simple
one
that
you
can
start
looking
at.
B
B
So
at
this
point,
I
will
wait
till
demo
here
so
on
the
left.
Here
we
have
a
command
window
and
what
we
can
do
is
we
can
ping
our
geodetesnet.io
like
I
mentioned
before,
and
you
know,
hopefully
it
returns
back
U.S
Central
One,
because
what
happens
that
the
DNS
resolve
to
the
closest
region,
whichever
closest
to
and
so
then
what's
really
interesting-
is
that
if
we
can,
we
can
use
a
VPN
clients
to
connect
to
a
Singapore
here
and
now
we're
pinging
back
from
Asia
Southeast
one.
B
B
B
The
traffic
actually
changes
across
the
world,
so
you
might
have
a
you
know:
a
central
peaking
where
Asia
Southeast
is
a
troughing
I
guess
so.
So
it's
really
important
to
actually
consider
like
more
than
just
geography.
It
could
be
time
of
day
and
things
like
that-
that
you
also
need
to
consider
and
respect
to
routing.
B
So
this
is
what
our
solution
for
Goa
DNS
routing
looks
like
right
now
we
are
using
Cloud
DNA,
Google,
Cloud
DNS
and
with
their
Geo
weighted
routing
policy,
and
this
gave
us
a
few
advantages
right
because
it
means
that
we're
a
bit
more
Cloud
native,
but
we
can
still
swap
out
the
that
DNS
solution
for
any
other
one
that
we
come
across
and
we're
at
the
end
of
the
day,
we're
just
running
kubernetes
on
Google
Cloud
engines.
So
it's
a
so
it's
also
something
we
could
swap
out
as
well.
B
B
So
we
can
return
to
our
earlier
definition
right,
so
Rebecca's,
iPhone
edition
of
a
distributed
system,
a
network
of
computers
working
together
to
appear
to
the
end
user
as
a
single
computer,
and
that's
like
a
very
classy
like
technical
or
like
classical
definition
of
what.
Maybe
a
distributed
system
might
be,
but
if
we
think
about
how
the
area
has
changed
right,
we
have
a
lot
more
people
on
their
phones
and
a
lot
more
users
who
are
not
a
part
of
the
internet,
like
maybe
10
years
ago.
B
B
B
So
the
way
to
think
about
this
is
kind
of
more
microservices
type
approach
where
you
think
about
all
the
data
that's
being
generated
across
the
world
in
terms
of
events
that
are
go
that
are
in
time.
So
these
events
can
be
ordered
and
they
can
be
pushed
to
Market
services
and
those
microservices
can
be
pushing
to
other
microservices
and
cues
and
things
so
you
might
be
knowing
What
kind
of
what
I'm
talking
about.
If
you've
used
any
of
these
tools,
some
message
queuing,
Pub,
sub
stream
processing.
A
A
The
next
thing
is
kind
of
geographic
event,
encoding
right
so
for
the
most
part,
Cloud
Solutions
assume
that
you
don't
want
to
know
about
geography
right,
like
that's
kind
of
what
the
cloud
is
right.
You
can
just
pretend
that
it's
this
magical
thing
up
here
and
it's
not
actually
physically
anywhere,
but
it's
actually.
It
is
physically
in
a
lot
of
places
and
having
that
Geographic
encoding
will
Empower
us
to
you
know,
have
a
better
user
experience
or
comply
with
local
regulations
where
the
data
is
right.
A
So
it's
going
to
kind
of
unlock
a
lot
of
features
that
we
need.
Another
one
is
total
ordering
by
default,
so
most
Eventing
Solutions
now
can
only
guarantee
total
ordering
for
Brokers
that
are
on
the
same
node
right,
and
that
is
because
total
ordering
is
expensive.
It
requires
consensus
right.
We
talked
about
that
at
the
beginning.
A
Strong
consistency
is
expensive,
it's
slower
and
you
actually,
you
have
to
pay
more
for
it,
but
if
we
had
total
ordering
as
kind
of
a
you
know,
readily
available
open
source
solution,
we
could
kind
of
guarantee
this
more
Equitable
Global
engagement
with
our
applications
and
then
the
last
one
that
I
assume
is
probably
most
exciting
to
a
group
of
kubernetes
people
is
automatic,
reconfiguration
in
fact,
I'm
hearing
a
lot
of
people
saying
those
words
automatic
reconfiguration
already
here
at
the
conference
this
week.
So
I
know
that.
That's
something
that's
on
everybody's
mind
right.
A
You
can't
just
infinitely
scale
your
devops
team
right.
We
need
better
algorithms
that
make
multi-cluster
more
realistic
for
small
teams,
which
probably
most
of
us,
work
on
a
small
team
and
not
like
a
huge
team
that
has
infinite
resources.
So
we
think
that
is
what's
coming
next.
If
you
want
to
check
out
a
little
bit
more
about
what
we
are
doing,
please
visit
us
at
rotational.app
or
just
come
and
chat
with
us.
A
A
Sorry
we
had
I
have
like
two
concurrent
questions
which,
like
perfectly
approves
the
point
that
you
know
failures,
happen
all
the
time
in
distributed
systems.
So
do
you
mind
if
I
answer
this
question
first
and
then
do
the
router
question?
Is
that
okay,
all
right?
So
you
said
we
built
our
own
distributed
system
and
yes,.
A
A
Yes,
great
question,
so
the
question
is
we
built
our
own
distributed
database?
Why
did
we
not
go
with
a
readily
available
commercial
solution
instead
of
doing
something
insane
like
building
our
own
distributed
database?
The
reason
is,
the
commercial
Solutions
are
super
expensive.
We're,
like
you,
know,
we're
working
for
a
non-profit
that
you
know
that
hired
us
to
do
this.
They
do
not
have
the
money
to
spend
on
like
Aurora
or
spanner,
or
something
like
that.
A
A
C
C
B
E
A
A
really
good
question,
so
the
question
is,
you
know
it
is
commonplace
in
geographically
and
distributed
systems
to
have
partitions
happen,
which
is
where
you
know,
two
parts
of
the
system
temporarily
can't
talk
to
each
other,
which
means
that
they're
making
a
lot
of
updates,
maybe
locally,
and
then
they
have
to
figure
out
how
to
synchronize
a
bunch
of
updates.
You
know
that
might
have
been
made
over.
A
You
know
an
hour
or
you
know,
maybe
even
longer,
if
the,
if
the
Falls
happens
for
a
long
time,
and
so
the
question
is,
you
know
in
our
solution,
how
do
we
kind
of
test
for
resiliency
and
recovery?
So
we
actually
have
run
some
experiment
comments
where
we
change
the
anti-entropy
interval
to
see
you
know
if
we
can
speed
up
the
anti-entropy
interval
right
so
like
speed
up
the
amount
of
synchronizations
that
have
that
happen,
we
reduce
staleness
and
it
does
help
protect
against.
A
You
know
having
to
do
a
lot
of
updates
if
there
is
a
partition,
but
we
don't
actually
have
an
experiment.
Yet
that's
orchestrated
to
kind
of
stimulate
like
a
earthquake
or
something
that
cuts
off
Singapore
from
let's
say
Ohio
or
something
like
that
where
our
other
you
are
our
U.S
server
is,
but
it's
a
really
good
idea
and
I
would
love
to
to
implement
that,
because
I
think
it
would
be
really
interesting
to
see
like
how
long,
basically
to
like
I
think
you
would
clock
it
right.
A
A
In
fact,
like
just
writing
everything
in
go,
you
know,
sort
of
always
encourages
you
to
think
about.
You
know
it'll,
probably
like
you
know,
if
they're
you
know,
if
they're
equals,
and
then
you
know
whatever
your
function
is
sort
of
the.
The
philosophy
is
like
this
probably
won't
work,
but
if
it
does
keep
going
and
so
kind
of
that
that
philosophy
of
just
kind
of
continually
checking
for
you
know
possible
errors.
You
know
we
do
have.
A
We
do
have
to
check
for
things
like
race
conditions
right,
because
that's
kind
of
a
very
common,
probably
way
more
common
than
earthquakes
for
sure,
so
that
yeah
is
a
great
question.
How
to
you
know
how
to
kind
of
have
that
conversation
about
what
to
test
for
I
would
say
you
know
concurrency
bugs
are
like
the
number
one
thing,
and
we
did
have
a
lot.
A
I
mean
it's
a
lot
of
like
the
anti-entropy
routine
is
like
extremely
concurrent
right
because,
as
one
replica
is
kind
of
sending
all
of
its
version
vectors
to
the
other
replica,
the
other
replica
is
starting
to
send
back
requests
for
updates
right
and
those
things
are
happening
concurrently.
So
there's
a
lot
of
things
that
can
go
wrong.
There
great
question.
F
A
So
it
was
like
people
who
are
going
between,
let's
say
like
South,
Korea
and
London
for
work,
and
they
would
get
recommendations
for
a
date
in
like
the
other
region
where
they
weren't,
and
so
it
is
something
that
I
think
you
know,
as
we
start
to
identify
these
bugs
people
are
going
to
start.
You
know
rethinking
kind
of
these,
these,
like
persistence
by
default,
total
ordering.
A
You
know
some
of
these
things
are
just
gonna
I
think
these
features
are
gonna,
become
more
commonplace
because
they
support
kind
of
a
way
better
user
experience,
and
it
hasn't
been
a
problem
until
recently,
because
you
know
stuff
was
just
not
fast
enough
for
that
to
ever
happen,
and
now
you
know
things
are
so
much
faster
that
we're
starting
to
see
a
lot
of
these
problems
crop
up
in
in
usually
in
user
interfaces.
That's
a
great
question.