►
From YouTube: Building Reliability with Spokes - GitHub Universe 2016
Description
Patrick Reynolds, Senior Systems Engineer, GitHub
GitHub hosts tens of millions of repositories on hundreds of file servers. The Git Infrastructure team runs Git as a distributed service, providing highly available and durable access to repositories, despite frequent hardware and software failures. One key part of the service is Spokes, a self-healing replication, routing, and load-balancing layer built on top of Git. In this talk by Patrick Reynolds, a Senior Systems Engineer at GitHub, he'll discuss how Spokes detects failures and routes around them to provide high availability. Hear about how Spokes uses real-time replication and voting to durably store repository content. Watch as we induce failures and, ideally, no one notices.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
A
Am
Patrick
Reynolds
I
run
the
get
infrastructure
team
at
github.
My
profile
picture
has
a
tiny
red
hat
I'm
wearing
this,
so
that
people
will
recognize
me,
but
now
I'm,
not
today,
I'm
going
to
talk
to
you
about
how
our
team
runs.
A
scalable,
reliable,
get
service
for
github
get
infrastructure
team
runs
the
get
service
at
github.
We
operate
two
layers
of
github
stack.
First,
we
run
the
file
servers.
We've
got
hundreds
of
get
file
servers
each
one
with
big
SSDs
lots
of
CPUs
and
a
few
hundred
thousand
repositories
above
that
layer
we
have
proxies.
A
A
We
have
internal
api's
that
the
rest
of
the
github
stack
uses
to
get
to
repositories
that
the
github
com
the
website
is
built
on
our
job
as
a
team
is
to
run
that
service
scale,
ibly
and
reliably.
We've
got
38
million
repositories.
You've
got
more
than
36
million
gist's.
Those
numbers
keep
growing
our
the
servers.
We
run
the
get
file
servers,
they
respond
to
over
a
billion
repository
touches
per
day.
A
So
some
of
these
are
simple
things
like
lists
the
branches
or
fetch
the
readme,
so
that
we
can
draw
the
front
page
of
the
repository,
but
a
lot
of
them
are
fetches.
Pushes
clones
there
they're
heavy
work,
it's
our
job
to
handle
all
of
that
traffic
and
to
handle
more
and
more
each
day
and
that's
our
job
to
make
it
reliable.
The
service
has
to
be
up.
The
service
has
to
not
lose
data
and
it
has
to
respond
every
one
of
those
requests
in
a
timely
fashion.
A
The
focus
of
my
talk
today
is
a
system
called
spokes.
Some
of
you
may
have
heard
of
this
previously,
as
d
get
the
name
is
now
spokes.
Spokes
is
the
distributed
system
that
maintains
to
get
repositories
on
those
file.
Servers.
Spokes
is
a
big
big
part
of
our
scalability
and
reliability
story
for
the
get
service
we've
been
working
on
spokes
for
coming
up
on
two
years
now,
and
at
this
point
we've
moved
every
repository.
Every
wiki
and
every
gist
so
that
it's
stored
in
spokes
the
goals
that
you
see
here.
A
These
are
the
goals
that
we
set
out
some
of
the
goals
that
we
set
out
when
we
decided
to
build
spokes
in
the
first
place.
These
are
things
that
the
previous
version
of
the
get
service
could
not
do
and
that
spokes
needed
to
and
now
can
so
briefly
anytime.
One
file
server
is
down.
Users
need
to
not
notice
that
there's
no
impact.
There
can't
be
publicly
obvious
when
I,
when
a
server
goes
down.
A
If
there
are
two
servers
down,
we
we
minimize
the
impact
and
I'll
get
into
more
detail
on
that
spokes
needs
to
automatically
balance
disk
space
and
cpu
load
so
that
there
just
aren't
hot
spots
in
the
cluster.
If
there
are
hot
spots,
they
need
to
be
self.
Resolving
spokes
needs
to
automatically
resynchronize
and
repair
any
after
after
any
damage.
So
failures
are
inevitable.
After
a
failure,
there's
there's
work
to
be
done.
A
Spokes
needs
to
figure
out
what
that
is,
and
do
it
automatically
spokes
needs
to
provide
horizontal
scaling,
meaning
we've
got
this
tier
of
github
stack
any
time.
We
need
more
capacity
on
that
tier
I.
Don't
want
to
do
anything
other
than
a
provision.
A
file
server
spokes
needs
to
do
the
rest,
automatically
maintenance
operations,
things
like
reboots,
taking
broken
servers,
offline
for
repair,
taking
old
servers,
aging
out
old
servers
permanently.
A
A
The
rest
of
my
talk
today
is
about
how
spokes
meets
these
goals,
there's
a
time-tested
way
to
achieve
reliability
in
a
distributed
system,
and
that's
replication
just
make
copies
of
stuff
if
your
stuff,
whatever
it
is,
easy
and
only
one
place,
then
it's
only
as
reliable
as
whatever
that
server
is
that
server
goes
down.
It
takes
your
content
with
it.
So
in
spokes
we
make
three
copies
of
everything:
spokes
spokes
this
application
level
replication
forget
we
make
three
copies
of
every
repository,
every
wiki
and
every
disk.
A
So
concretely,
what
this
means
is
when
you
create
a
new
repository,
we
actually
create
three
physical
git
repositories
on
disk
on
three
different
servers
and
every
time
you
do
some
sort
of
change
that
repository
or
write
a
push,
opening
or
closing
a
pull
request.
We
write
that
synchronously
in
three
places.
We
update
all
three
servers
together
now
this
is
a
little
bit
unusual.
Most
distributed
systems,
do
their
replication
at
the
layer
at
the
level
of
entire
disks
or
entire
servers.
So,
for
example,
raid
says,
make
a
copy
of
this
entire
disk.
A
If
one
disk
goes
away,
we
can
rebuild
it,
and
my
sequel
typically
replicates
entire
servers.
Previously,
our
get
service
used
a
product
called
drbd
that
also
replicates
entire
servers.
Spokes
does
not
not
here
so
spokes
is
replicating
individual
repositories,
which
gives
us
enormously
more
flexibility.
Here's
how
that
works.
So
every
right
goes
to
all
three
repositories
at
once
and
as
long
as
two
repositories,
except
to
the
right,
they
come
up
with
the
same
answer
after
the
right.
We
consider
it
a
success.
Every
red
goes
to
just
one
repository.
A
There's
there's
no
reason
to
replicate
a
read
on
which
means
that,
as
long
as
any
one
of
the
three
copies
of
something
is
up
reads
can
still
succeed
now
I've
said
the
number
three
here
are
several
times
we
make
three
copies.
That's
not
hard
coded!
That's
a
policy
choice.
We
chose
three
because
it's
the
smallest
number
of
replicas
that
can
tolerate
one
failure
and
still
commit
rights,
but
if
we
had
reliability
requirements
that
said
four
copies,
five
copies
six
copies.
Obviously
that's
the
thing
we
can
do
so.
A
A
Oh
now,
when,
when
github
started
out
and
I
think
we're
a
lot
of
sites
start
out
is
when
you
have
a
small
number
of
servers,
you
can
pretend
they
don't
fail
and
you
can
just
apologize
nicely
when
they
do.
When
you
have
a
few
more
servers,
you
can
spend
lots
of
money
and
you
can
buy
the
most
reliable
hardware.
A
You
possibly
can
get
everything
redundant,
redundant,
disks,
redundant,
networking,
redundant
power
supplies,
ECC
memory
and
for
a
while
you
can
keep
pretending
that
failure
isn't
real,
but
when
you've
got
hundreds
of
servers,
the
illusion
breaks
down
I
mean
even
we,
we
have
expensive,
very
reliable
servers
and
they
still
do
fail.
We
lose
servers
enough
that
this
matters.
You
can
see
this
here.
This
is
a
screenshot
in
the
middle
of
this
slide,
that's
a
screenshot
from
the
github
Status
page
from
last
December.
A
This
is
the
old
file
server
cluster,
not
spokes,
and
so
reading
bottom
to
top
at
303
p.m.
we
are
troubleshooting
an
issue
affecting
a
small
number
of
repositories.
Then
in
17
minutes
later
we
are
continuing
to
investigate
the
issue.
Less
than
five
percent
of
repositories
are
affected
and
then,
finally,
eight
minutes
after
that
everything
is
operating
normally.
Now
on
this
old
cluster,
less
than
five
percent
of
repositories,
that's
code
for
a
file
server
has
crashed.
A
We
no
longer
have
status
messages
like
this
file
servers
when
they
crash
we
no
longer
have
to
have
to
tell
status,
but
you
can
see
here
it
took
25
minutes.
It
was
25
minutes
of
status
just
because
one
file
server
crashed,
which
is
not
supposed
to
be
a
big
deal.
That's
how
long
it
took
for
a
human
being
to
notice
to
manually
failover
the
server
to
reboot
for
every
for
us
to
verify
that
everything
was
healthy
and
then
to
notify
the
world
that
we're
back
to
normal.
A
Now
that
we
have
three
copies
of
everything
we
just
route
around
failures
like
this,
we
instead
we
send
every
request
to
the
best
server
for
handling
that
request,
and
so
the
question
is:
how
do
we
know
it's
best?
We
start
with
a
list
of
replicas,
so,
as
I
said,
every
every
repository
is
replicated
in
three
places.
We
filter
that
replica
list,
depending
to
which
replicas
are
online,
not
overloaded
and
up-to-date,
and
ideally
out
of
this
filtering
step,
we
still
have
three
servers.
A
A
We
try
the
first
server
in
the
list.
The
next
server
the
next
server
one
at
a
time.
Ideally
the
first
one
should
succeed
with
a
right.
We
send
it
to
all
three
and
as
long
as
two
of
them
come
back
with
the
same
answer.
Yes,
I've
accepted
this
right
here
is
the
result.
Then
we
consider
the
right
of
success
and
we
tell
the
client
that
the
right
succeeded.
This
still
raises
the
question:
how
do
we
know
which
servers
are
online?
A
That's
part
of
the
filtering
step
is:
is
online
replicas
only
and
the
way
we
the
way
we
do
that
we
have
a
failure.
Detector
specifically
spokes,
watches
real
application
traffic.
It
watches
every
internal
API
call
it
watches
every
fetch,
push
clone
and
anytime.
We
see
three
failures
in
a
row
to
the
same
server
we
that
server
is
offline.
As
soon
as
we
see
a
single
success,
we
mark
the
server
back
online.
A
Now
it's
important
here
that
we
watch
real
application
traffic,
so
a
simpler
solution
would
be
just
to
have
heart
beats,
but
heartbeats
aren't
very
useful.
They
don't
test
much.
So,
for
example,
the
simplest
heartbeat
is
ping
and
all
the
ping
is
actually
testing
is.
Does
the
remote
end
still
have
a
colonel
with
a
working
networking
card
because
you're
not
even
going
to
user
space
when
you're
pinging,
something
all
you're
asking
is?
Do
you
have
an
ICMP
stack
which
is
not
a
great
approximation
of?
Can
you
handle
user
to
handle
application
traffic
next
level?
Up?
A
Maybe
you
maybe
have
a
heartbeat
that
makes
TCP
connections
and
in
fact
we
have
internal
API
called
online,
and
all
this
does
is
make
a
TCP
connection
and
get
back
true.
The
serialized
value.
True
and
again,
you're
still
not
testing
very
much
you
you
can
design
better
and
better
heartbeats.
You
could
design
a
heartbeat.
That
says
here
is
a
repository
that
has
to
be
up
and
we
send
ghost
traffic
to
it,
and-
and
you
know
what
the
right
answer
is
for
this
repository.
In
fact,
all
you're
still
really
testing
is
that
repository.
A
There
are
still
failure
modes
like
it
disk
is
down
or
the
get
binary
is
corrupt
or
a
system.
Library
is
go
on
where
you've
got
failure
modes
that
none
of
these
heartbeats
are
going
to
detect,
but
the
thing
that
can
detect
does
application.
Traffic
work
is
the
application
traffic
itself,
so
our
failure
detectors
is
based
entirely
on
that
we
do
use.
Heart
beats
for
two
things.
Once
we've
marked
a
server
offline,
we
stop
sending
it
application
traffic.
A
So
the
only
way
to
know
when
it's
come
back
online
is
with
a
heartbeat,
and
so
the
heartbeat
has
a
prospective
yes.
This
is
now
responding
to
TCP
connections,
and
then
we
go
back
to
using
the
application
traffic
to
say:
is
it
really
back
online?
The
heartbeat
also
pulls
the
the
cpu
load
for
the
server
and
I'll
come
back
to
that
in
four
slides.
That's
important
for
slides
from
now
now
requiring
three
failures
to
detect
the
server
is
offline
still
means
that
three
user
visible
operations
failed
and
our
goal
here
was
zero.
A
So
what
we
have
as
I
as
I
said
the
the
routing
process
that
filters
and
sorts
the
list
of
replicas
the
output
of
that
is
still
a
list
of
replicas,
even
though
we
only
need
one.
We
provide
three,
and
so
everything
that
performs
reads:
fetches,
fetches
clones
and
all
of
the
internal
AP.
Is
that
look
like
reads?
A
A
One
key
fact
about
our
failure:
detector
is
it
actually
maintains
a
separate
list
of
failed
nodes
from
the
point
of
view
of
each
client?
Now,
why
do
we
do
that
when
one
file
server
is
offline?
So
here
you
can
see
we
are
sending
application
traffic
to
DFS
for
sorry,
take
a
step
back
the
nodes
on
top
here.
These
are
the
clients
of
our
internal
API
and
the
nodes
on
the
bottom.
A
They've
all
detected
the
correct
truth,
so
we
could,
in
this
case,
have
a
single
master
list
that
says:
file
server
for
is
down,
but
not
here
what
happens
if
one
of
the
clients
is
faulty
in
its
implementation
of
the
failure
detector
or
if
one
of
the
clients
loses
access
to
the
network
that
goes
to
the
file
servers,
but
can
still
report
to
the
failure
detector
that
it
thinks
every
file
server
is
down.
You
don't
want
to
amplify
this
and
say
old.
A
A
A
All
right,
we're
logged
into
production
here
and
I'm
going
to
find
servers
that
are
not
in
the
middle
of
long-running
clones
and
I'll.
I'll,
explain
why,
in
a
minute,
let's
reboot,
DFS,
1174
and
just
for
fun,
that's
a
typo
1174,
let's
get
a
console
for
1174
so
that
we
can
watch
so
that
you
know
that
this
is
not
fake.
A
All
right
there
is
a
client,
it's
got
a
root
prompt
and
now
we
will
this
being
github.
We
reboot
things
with
chat,
ops,
so
G
panel
host,
reboot,
github,
DFS
1174
the
whole
fully
qualified
name
and
Hugh
bot
wants
me
to
say
a
magic
word
just
so
that
I
don't
do
this
by
mistakes,
not
a
security
measure.
It's
just
a
typo
measure.
Take
this
moment
to
make
sure
your
rebooting.
The
right
thing.
A
This
is
our
heart
would
not
a
heartbeat
sorry
or
failure
detector.
This
is
the
RPC
calls.
Oh
look
at
that.
That's
a
lot
of
them
failing,
which
is
exactly
what
we
want
to
see.
This
is
a
measure
of
actual
application
traffic
that
is
failing
off
of
DFS
1170
for
and
on
to
something
else,
there's
a
massive
spike
here.
This
is
the
graph
that
we
base
our
alerts
on,
because
you
can
see
it's
a
very
clean
graph
except
that
suddenly
DFS
1174
has
this
huge
spike
of
hundreds
of
requests
that
suddenly
had
to
failover.
B
A
A
Well,
I
hope
not
when
when
a
server
is
down
or
failure,
detector
notices
it
and
we
route
traffic
away
from
it
in
real
time,
any
traffic
that
was
supposed
to
go
to
that
server
just
fails
over
to
the
other
replicas
quickly,
so
that
no
user
visible
operations
fail
and
because
we're
replicating
at
the
level
of
repositories.
This
doesn't
even
create
hotspots.
A
So
if
we
were
replicating
entire
servers,
when
you
take
one
down,
it
would
have
exactly
two
servers
available
to
pick
up
the
slack,
but
because
we're
replicating
at
the
level
of
repositories,
the
rest
of
the
cluster
now
proportionally
picks
up
the
slack.
So
we
haven't
created
a
hotspot.
So
indeed
this
is.
This
is
no
impact
and
then
writes
go
to
all
healthy
servers.
So
because
we've
taken
one
down,
everybody
still
has
at
least
two
copies.
All
the
rights
are
still
succeeding.
A
Our
next
goal
was
that
when
two
servers
go
down,
there's
minimal
impact,
and
so
the
best-case
scenario
here
is
two
servers:
go
down
in
the
same
rack
and
there's
no
impact
at
all,
and
that's
because
we
treat
racks
as
availability
zones.
Racks
servers
within
Iraq
have
somewhat
correlated
failures.
You
can
lose
power
to
that
rack.
You
can
lose
network
to
the
rack.
Somebody
really
strong
could
come
and
tip
over
the
entire
rack,
whatever
whatever
it
takes,
those
those
servers.
A
The
servers
in
the
rack
are
somewhat
more
correlated
than
servers
in
separate
racks,
so
we've
built
in
this
assumption
that
we
can
lose
an
entire
rack
and
there's
no
user
impact,
but
if
we
lose
two
servers
in
two
different
acts
right
now,
it's
about
one
one-hundredth
of
a
percent
of
repositories
that
are
now
read-only.
They
don't
accept
rights
until
at
least
one
of
those
servers
comes
back
up
and,
interestingly,
that
percentage
actually
drops,
as
our
cluster
gets
bigger
and
bigger.
That's
given
the
current
size
of
the
cluster,
that's
a
tiny
percentage.
A
A
The
third
goal
we
can
look
at
now
is
balancing
cpu
load,
and
this
is
that
this
turns
out
to
be
really
easy.
You
treat
remember
I,
said
four
slides
from
now.
This
is
four
slides
from
now
you
treat
any
server
with
an
unusually
high
system
load
as
offline,
even
though
the
server
is
still
present.
You
just
stop
routing
traffic
to
it.
You
proportionally
spread
that
traffic
over
the
rest
of
the
cluster
until
that
server
can
work
through
whatever
that
backlog
is,
and
previously
we
used
to
fail
servers
in
the
old
file
server
cluster.
A
We
used
to
fail
servers
just
because
they
were
overloaded,
they
would
hit
a
system
load
of
80
or
something
and
they
would
start
thrashing
and
they
would
be
unable
to
keep
up
with
a
load
arriving
on
that
server.
Now,
when
you
had
run
a
service,
it
hits
a
system
load
of
80.
We
just
stopped
sending
things
to
it
until
it
works
through
its
backlog,
and
it
stops
thrashing
this.
A
This
feature
also
is
a
crude
way
to
allow
a
single
repository,
the
capacity
of
three
full
servers,
so
it
fills
up
one
server
and
it
spills
over
onto
another
server
and
it
spills
over
onto
another
server.
We
have
a
small
number
of
very
busy
repositories
that
do
actually
do
that,
shifting
gears
a
little
bit
what
happens
after
a
downtime
and
what
happens
after
a
failure.
A
So
if
a
server
is
offline
for
a
while
we're
accepting
hundreds
of
writes
per
second
not
to
an
individual
server
but
across
the
cluster,
but
that
still
means
that,
if
we're
down
four
minutes,
we
have
dozens
or
hundreds
of
rights
intended
for
that
server
that
didn't
go
to
that
server.
So
this
is
this
is
standard,
replication
question:
how
do
you
catch
it
back
up
after
it,
after
it
recovers
and
then,
after
a
failure
that
server
is
not
coming
back
after
a
failure?
A
You've
got
to
replicate
everything
that
was
on
a
server
that
failed,
so
it
had
hundreds
of
thousands
of
repositories.
Now,
every
one
of
those
repositories
is
down
to
two
copies:
we
have
to
create
a
third
copy
somewhere
else.
Two
copies
is
okay.
The
problem
with
two
copies
is
any
other
failure
in
the
cluster
the
unless
it's
in
the
same
rack,
any
other
failure
in
the
cluster
is
suddenly
going
to
be
user
visible.
A
A
Goals
are
properties
that
are
supposed
to
be
true,
they're,
not
invariants.
They
are
properties
that
sometimes
aren't
true
but
they're
their
properties
that
are
supposed
to
be
true
and
we
express
them
as
sequel
queries
and
these
these
sequel
queries
return
a
list
of
repositories
for
which
the
goal
isn't
true
and
I'll
in
a
moment,
I'll
get
into
some
specific
examples.
Repairs
are
paired
with
goals
for
each
goal.
There
is
a
repair
which
is
a
small
ruby
program
that
can
make
that
goal.
A
True,
if
it
isn't
already,
and
so
anytime,
we
find
a
goal
that
isn't
met.
We
schedule
the
appropriate
repair
using
rescue,
which
is
github
job
queue
open
source.
If
you
want
it
goals,
all
of
our
goals
are
checked
every
two
minutes
and
then
the
repairs
run
asynchronously
after
that.
So
here
are
three
concrete
goals
on
every
the
replicas
of
every
repository
are
supposed
to
match.
A
So
if
you
make
three
copies
of
things,
they
should
have
the
same
contents
and
we
check
every
two
minutes,
which
sounds
really
expensive,
but
in
fact
what
we're
checking
is
checksums
we
take
when,
whenever
we
write
to
a
repository,
we
update
a
checksum.
So
what
we're
looking
at
is
do
the
check
sums
of
the
three
replicas
match
and
any
time
that
they're,
not
we
schedule
a
repair.
The
repair
just
says
copy
the
bits
from
a
copy
that,
from
a
replica
that's
healthy
to
a
rep
to
the
replica
that
isn't
second
goal.
A
A
Finally,
we
just
have
it:
we
have
as
a
goal
or
all
the
disks
balanced
which
looks
at
the
how
full
all
the
disks
are,
and
if
there
is
more
than
ten
percent
between
the
fullest
disk
and
the
emptiest
disk,
we
move
replicas
around
and
this
this
is
another
great
flexibility
of
replicating
at
the
level
of
repositories
is
UK.
It's
granular.
You
can
move
a
little
bit
from
the
fullest
server
to
the
emptiest
server
and
you
just
keep
repeating
that
process
until
the
disks
are
sufficiently
balanced.
A
A
This
is
this
is
inefficient
in
that
at
some
point
you
knew
you
had
a
handful
of
needles
and
you
know
that
you're,
the
one
throwing
those
needles
that
there
is
a
point
in
the
code
where
we
could
detect
the
error
conditions
and
we
could
try
to
schedule
the
right
repair
at
that
moment,
but
we
don't-
and
so
this
this
slide
is
how
I
justify
why
first
I
this
goal
based
programming,
it's
decoupled
in
code,
so
keeping
the
repair
mechanisms
separate
from
product
code
makes
both
of
them
simpler.
Our
team.
A
They
get
infrastructure
team
that
built
spokes,
didn't
write
the
product
code.
There
is
eight
years
nine
years
of
github
github
product
code.
That
runs
the
website.
We
didn't
write
most
of
it,
we
didn't
read
all
of
it,
but
we
still
have
to
we've
introduced
new
error
pads
into
that
code,
and
so,
rather
than
go
back
and
put
rescue
statements
in
all
of
that
code
and
try
to
catch
those
errors
and
repair
them
in
real
time.
A
Second,
is
it's
decoupled
in
time
when
a
failure
happens,
turns
out
to
be
a
lousy
time
to
try
to
fix
that
failure.
The
failure
probably
happened
because
the
server
was
down
or
is
overloaded
or
a
process
crashed
or
something
timed
out.
This
is
not
a
good
time
to
say:
well,
let's
copy
some
more
bits
around
you're,
probably
going
to
make
you
either
either
the
system
is
unavailable
enough
that
you
can't
run
the
repair
or
it's
available,
but
overloaded
you're
just
going
to
make
the
problem
worse,
so
we
don't.
A
Instead,
we
have
this
process
that
two
minutes
later
comes
and
says:
what's
broken,
and
can
we
fix
it
now
and
if
we
can't
then
two
minutes
later,
it's
still
broken
and
you
ask
again
and
you
and
you
reschedule
the
repair,
and
so
you
just
automatically
keep
trying
these
repairs
until
you
can
and
until
they
can
succeed
and
they're
throttled
as
well.
If
there
are
too
many
repairs
for
a
two-minute
period,
they'll
still
be
broken
two
minutes
later,
that
repair
will
get
rescheduled.
A
Finally,
this
this
code
is
reusable,
so
you
can
tinker
with
the
goals
to
accomplish
maintenance
tasks.
Let's
say
we
wanted
four
copies
of
everything.
We
set
a
goal
that
there
are
four
copies
of
everything.
Nothing
else
changes.
We
don't
have
to
write
the
scripts
to
make
those
copies.
We
just
say
now,
there's
supposed
to
be
four
and
the
task
that
creates
more
copies
of
replicas
just
goes
and
does
it
it
will
probably
take
days.
We
have
a
lot
of
repositories.
A
A
We
had
as
a
goal:
balance
disk
space
and
and
and
as
I
said,
the
way
we
do
this
is
we
set
a
repair
goal
which
is
distinct
from
our
design
goals.
We
said
as
a
repair
goal.
Disks
are
balanced
and
we
have
a
repair
process
that
moves
repositories.
So,
yes,
we
did
this
another
design
goal
we
automatically
resync
and
repair
after
any
sort
of
server
level
failure.
And
yes,
we
created
a
repair
goal
that
said
repository
contents.
A
A
So
if
somebody
is
spending
12
hours,
downloading
cyanogen
over
some
very
slow
link,
you
don't
want
to
10
hours
into
that
say
that
was
a
good
time
to
reboot
the
server,
because
they're
gonna
have
to
start
again
from
scratch,
and
so
it's
just
rude
to
have
a
planned
maintenance
that
involves
cutting
off
these
long
running
downloads.
So
instead
we
flip
a
bit
that
says
this
server
is
going
away
soon,
don't
start
any
new
work
on
it.
A
Similarly,
with
retiring
a
server
you
set
a
bit
on
that
server.
That
says
this
server
is
going
to
retire
soon,
evacuate
all
of
the
replicas
from
it
in
a
in
a
controlled
fashion,
and
so
this
this
gets
back
to
the
goal
based
programming.
You
just
set
a
goal
that
says
I
the
replicas
on
the
server
that's
going
away:
they
don't
count
anymore.
So
now
there
are
only
two:
if
you
need
those,
you
can
go
get
them.
But
in
terms
of
do
we
have
three
copies,
those
don't
count.
A
We
have
only
two
copies
of
all
of
thousands
of
repositories.
Now,
it's
time
to
fix
them,
and
then
you
give
it
a
day
and
it
makes
a
third
copy
of
everything.
And
now
the
server
is
empty
and
you
can
get
rid
of
it
and
definitely
nobody
cares.
The
reason,
as
I
said.
The
reason
you
don't
want
to
get
down
to
two
copies
of
things
is
that's
healthy,
but
it
doesn't
tolerate
another
failure.
You
don't
want
to
risk
having
that
failure
during
the
day
that
it
takes
to
rebuild
the
server's
replica
somewhere
else.
A
A
A
So
there's
the
previous
spike
you
can
see.
Dfs
1174
was
down
for
three
and
a
half
for
four
minutes,
but
now
it
came
back.
I
mean
there.
There
are
no
more
failures
to
it,
there's
nothing
more
to
route
and
we're
just
flatlined.
All
the
servers
are
healthy
again,
so
that's
good
news
doing
that
demo
and
it
didn't
break
things
back
to
the
talk.
So
the
final,
the
final
technical
goal
here
was
horizontal
scaling.
A
We
want
to
be
able
to
add
more
CPU
or
disk
space
and
have
spokes
just
automatically
take
advantage
of
it
and
step
one
add
a
server.
There
is
no
step
to
the
spokes
since
spokes
automatically
balances
the
disks.
When
you
add
a
server
that
server
is
empty
and
that's
out
of
that
that
doesn't
satisfy
the
goal.
A
So
if
you've
heard
of
enterprise
cluster
spokes
is
how
we
manage
all
the
file
servers
in
enterprise
cluster
all
the
same
technologies,
lessons
I
hope
that
you
take
away
from
this
talk.
One
replicate
everything
that's
important
this.
This
is
basic.
If
you,
if
you
care
about
data,
make
more
than
one
copy
of
it,
somehow,
whether
that's
raid
or
drbd,
or
database
replication
or
building
your
own
bespoke
application
level
replication
solution
whatever
it
is,
just
make
sure
you
make
enough
copies,
de-emphasize
individual
servers.
A
So
if
you've
built
a
distributed
system-
and
you
can
point
to
one
server
and
say
I
will
get
woken
up
in
the
middle
of
the
night.
If
that
server
fails,
your
design
isn't
right.
Yet
application
traffic
is
better
for
detecting
failure.
Then
heartbeats
our
heartbeats,
don't
test
anything
important.
If
you're
trying
to
answer
the
question,
can
this
server
responds
to
application
traffic?
The
best
way
is
to
look
at
the
return
codes
from
actual
application
traffic.
A
And,
finally,
if
if
this
talk
was
interesting
to
you
and
I
hope
that
it
was,
we've
got
a
couple
of
blog
posts
up
about
it
at
github
engineering
com,
lots
of
other
blog
posts
too,
but
two
of
them
specifically
on
spokes
and
that's
the
end
of
the
talk.
Thank
you
for
your
attention.
I'd
be
delighted
to
take
questions.
A
A
There's
also,
we
have
the
the
spillover
mechanism
when
some
server
gets
massively
overloaded.
We
also
already
today
can
flip
a
switch
for
individual
repositories.
That
say
just
load
balance
this
one
always
so
that
that
step
that
says,
find
the
replicas
filter,
it
sort
it.
You
can
put
a
random
number
in
the
sorting
routine,
so
that
reads:
go
randomly
to
three
copies
and
you
can
even
wait
it
so
that
if
one
of
the
servers
happens
to
be
bigger-
and
this
this
works
really
well,
that's
for
three!
So
far
we
could
certainly
go
higher
numbers
later.
C
A
Absolutely
so
by
default,
the
failure
is
sorry,
the
repairs
are
locked
free,
which
means
they
are
intentionally
copying
from
a
target
that
could
have
a
failure
or
even
accept
a
right,
and
you
sort
of
get
halfway
through
that
right.
What
happens
is
at
the
end
of
a
repair
we
recalculate
the
checksums
atomically
on
all
three
and
if
they're
still
in
disagreement,
then
we're
going
to
do
another
repair
and
after
a
sufficient
number
of
failed
repairs,
then
we
actually
take
a
lock
and
we
say
no
changes
until
this
gets
properly
repaired.
A
E
A
F
A
A
So
when
they're
not
unanimous,
we
still
have
one
that
all
of
the
other
copies
are
supposed
to
agree
with,
and
so
as
long
as
the
definitive
checksum
agrees
with
one
of
them,
the
other
two
can
catch
up
to
it.
It's
not
always
a
majority
vote.
There
are
cases
where
it's
these
to
return
an
error
in
this
one
return
to
success,
for
example,
3
3
turns
out
to
be
enough.
We
very
rarely
have
to
step
in
and
manually
repair
anything
and
and
when
we
do,
the
manual
repair
isn't
repo
surgery.
It's
just.