►
From YouTube: Scaling at GitHub, Patrick Reynolds - Git Merge 2016
Description
The GitHub Systems team provides Git as a fast and reliable service, both as the backend for GitHub.com the website, and directly to users. To do that, we first make Git itself fast and reliable. But we also must consider Git as a distributed application, at the scale of billions of operations per day. Hear about the challenges that presents, and learn about the bespoke tools GitHub has built to monitor, shape, cache, and replicate transactions in our Git service.
Note: In this talk, Patrick refers to DGit, which was an internal codename for the project to distribute the storage of git repositories more efficiently within GitHub's infrastructure. In order to reduce redundancy and potential confusion, GitHub has subsequently renamed the project Spokes.
Patrick Reynolds leads the team at GitHub that runs Git as a service. He has been building tools to monitor and troubleshoot distributed applications for more than ten years: first as an academic research project, then as a product at BlueStripe Software, and most recently for internal use at GitHub.
A
How
we
run
our
github
repository
file
servers
so
that
we
can
host
that
many
Republic
of
repositories
efficiently
and
reliably
at
scale.
So
the
motivation
here
github
is
big.
Github
got
about
35
million
repositories
about
30
million
gist's,
it's
growing
fast.
We
execute
some
form
of
git
or
Lib,
get
to
over
a
billion
times
a
day
on
those
repositories
and
in
spite
of
all
that,
github
needs
to
be
fast
needs
to
be
reliable.
It
needs
to
be
consistently
fast.
A
So,
for
example,
our
99th
percentile
page
load
times
on
the
website
are
about
half
a
second,
which
means
a
lot
of
those
those
page
builds.
They
go
to
get
repositories,
they're,
calling
yet
they're
shelling
out
to
get
or
they're
hitting
lib
get
to
the
library
and
so
to
get
that
page
built
in
half
a
second
those
touch,
those
repository
touches.
They
need
to
come
in
well
under
half
a
second,
at
least
on
at
least
99%
of
the
time,
and
in
addition
to
this,
when
you've
got
35
million
repositories,
you
see
a
lot
of
corner
cases.
A
We
see
really
big
repositories.
We
ask
people
to
keep
the
repositories
under
a
gig
some,
sometimes
they
go
over
40
gigs
anyway.
We
reserve
the
right
to
lock
them,
but
sometimes
people
do
it.
We've
got
some
very
deep
commits
histories.
We've
got
repositories,
they
go
back
20
years,
which
obviously
predates
get.
So
you
look
in
the
history.
It's
CBS
for
a
while
subversion
for
a
while.
Yet
for
a
while
and
glad
they
converted,
we've
got
repositories
that
go
back.
500,000
commits
Linux
is
one
of
them.
It's
not
the
only
one
of
them.
A
That's
got
a
history
that
deep
get
log,
get
blame
or
pretty
expensive
when
you've
got
a
history
that
deep
we've
got
some
repositories
with
as
many
as
a
million
branches
and
tags,
we've
got
repositories
with
high
push
rates,
so
we've
got
some
repositories
that
push
ten
thousand
times
a
day
effectively
using
github
as
a
log
file,
just
every
five
seconds
or
so
during
the
business
day.
And
then
we've
got
some
public
repositories
too.
A
We've
got
a
public
repository
with
80,000
Forks
we've
got
a
public
repository
that
routinely
gets,
gets
requests
from
a
hundred
thousand
different
client
IPS
I'll
get
a
little
bit
more
into
that
last
one
later
in
the
talk.
So
the
rest
of
this.
The
rest
of
this
talk
I'm
going
to
talk
about
how
we
run
get
at
scale
and
that
breaks
down
into
four
things.
One
is
get
itself
has
to
be
fast.
A
If
you
double
the
speed
of
something
and
get,
then
you've
doubled
the
amount
of
repositories
that
you
can
run
on
your
servers,
so
any
any
improvement
to
get
is
has
a
huge
impact
both
on
the
reliability
of
the
site,
the
usability
of
the
site
and
how
many
repositories
we
can
host
the
second
one
flipside
of
that.
Sometimes
we
have
to
slow
get
down
the
third.
A
A
So
the
first
thing
this
is
this
is
our
workflow
for
making
yet
fast,
as
I
said,
any
improvement
to
get
has
a
huge
impact
to
github
as
a
whole.
We
have
to
monitor.
We
have
hundreds
of
repository
servers.
We
monitor
them
with
graphite.
We
have.
We
have
alerts
on
a
bunch
of
those
metrics,
so
any
server
where
the
CPU
is
overloaded
or
it's
running
out
of
memory
or
response
time
is
getting
too
slow.
A
We
get
an
alert,
and
that's
that's
really
the
first
indication
that
something's
gone
wrong,
that
we
need
to
go
investigate
the
the
second
is
triage.
So
once
you
know
what
server
has
gone
wrong,
some
particular
file
server
is:
has
two
highest
CPU
load
or
calls
to
get
core
are
taking
too
long.
Now
on
that
server,
you
want
to
find
out
what's
actually
happening
on
that
server,
so
we
log
every
single
get
operation
anytime.
A
We
shell
out
to
get
we
we
log
to
disk
the
repository,
the
command
that
was
being
run,
the
user
ID,
the
client,
IP
and
then
performance
things
like.
How
long
did
it
take?
How
much
CPU
time
did
it
take
so
when
a
server
starts
to
suffer
for
some
reason,
it's
pretty
easy
to
go
in
and
say
what,
on
that,
server
is
consuming
the
most
resources
right
now.
A
Then
you
have
to
recreate
it
and
so
those
those
logs
that
we
keep
have
a
very
small
amount
of
information
about
each
each
request,
because
we
get
a
lot
of
requests,
but
once
we've
narrowed
it
down
to
a
problem,
we
can
turn
on
more
detailed
logging.
We
turn
on
basically
everything
that
the
client
sends
us
in
the
requests
that
we
can
replay
it
capture
it
in
a
debugger
capture
in
a
profiler
and
find
out
what's
actually
wrong
and
then
fix
it.
A
Maybe
the
rest
of
the
talk
is
about
what
happens
when
we
don't
or
can't
fix
the
particular
bottleneck,
but
our
goal
is
to
get
any
any
call
to
get
core
under
about
a
second
and
that's
the
goal.
We
don't
hit
100%
of
the
time,
but
we
hit
very
close
to
hundred
percent
of
the
time.
The
rest
of
the
talk
is
about
that
fraction
of
a
1%
when
we
fix
bottlenecks
and
get
we
upstream
them.
So
here
is
a
an
arbitrary
sample
of
bottlenecks
that
we've
found
and
fixed.
Often
it's
something
that's
quadratic.
A
So
it's
something
that
takes
quadratic
CPU
time
in
the
number
of
references
or
in
the
number
of
remotes
or
in
the
number
in
the
commit
history
or
anything,
and
it's
just
a
matter
of
finding
that
line
of
code
and
converting
an
array
to
a
hash
table
or
whatever
the
the
biggest
one
here
at
the
bottom,
which
is
way
bigger
than
converting
an
array
to
a
hash
table
was
was
reach
ability
bitmaps.
This
is
something
that
VMG
on
the
systems
team
at
github
did
based
on
work
on
J
get
at
Google.
A
They
Google
invented
bitmaps
vmg
wrote
an
implementation
for
the
Korg
yet
for
the
C
implementation
of
get,
and
you
can
see
in
the
graph
on
the
right
we
started
out.
This
is
how
long
it
used
to
take
to
clone
the
Linux
kernel
it
took
about
a
minute
and
after
bitmaps
it
took
about
a
second,
and
so
that's
that's
huge.
That's
that's
very
close
to
our
goal
of
getting
everything
down
under
a
second.
A
It's
a
set,
it's
1.6
seconds
of
CPU
time
to
clone
the
Linux
kernel,
and
this
made
the
file
server
at
the
time
one
file
server
that
we
had
hosting
the
Linux
kernel
way
happier
way.
We
use
way
less
CPU,
but
we
can't
always
make
individual
get
operations
fast.
Some
of
them
are
still
going
to
be
slow
and
also
there's
some
that
we
just
haven't
made
fast.
A
Yet
we
get
we,
we
get
new
corner
cases
every
day
we
get
new
surprises,
and
so
until
we
find
a
way
to
make
that
fast
or
unless
we
can
find
a
way
to
make
that
fast,
we
need
to
prioritize
and
delay
get
operations.
So
there
are
two
graphs
here.
These
are
old
old
graphs,
because
the
system
I'm
talking
about
here
we've
had
an
operation
for
about
two
years,
but
on
the
Left,
this
is
CPU
load
on
on
a
file
server.
A
So
the
green,
you
know
steady,
steady,
steady
and
then
boom
a
bunch
of
requests
for
the
same
repo,
basically
identical
requests
for
the
same
repo
come
in
and
suddenly
the
server
is
slammed.
The
blue
line
is
that
servers
hot
spare
here?
This
is
memory
on
that
server
and
red
is
memory,
mapped
processes
and
the
the
bluish
is
is
block
cache
yet
is
a
or
get
servers
or
basically
file
servers.
A
They
need
lots
of
block
cache
so
once
that
block
cache
gets
squeezed
out
this
machine
thrashes
itself
to
death
and
I,
don't
know
if
those
of
you
in
the
back
can
see
it,
but
there's
a
little
gap
here.
That's
where
the
machine
crashed,
and
here
I
said
the
blue
is
the
hot
spare.
That's
where
the
hot
spare
took
over.
We
used
to
get
these
pretty
regularly.
A
We
would
get
a
load
spike
and
we
would
lose
a
file
server
and
get
Mon
is
the
system
we
built
to
fix
that
and
I'm
gonna
talk
a
little
bit
about
how
Gabon
works
game
on
sets
limits
for
how
much
CPU
memory
and
network
any
given
user
can
use
these?
If
they're
really
generous
limits,
it's
basically
impossible
for
individual
developers,
even
a
whole
team
of
individual
developers
to
hit
this
limit.
The
only
thing
that
hits
this
limit
is
scripts
and
we've
got
lots
of
scripts
hitting
github
cloning
for
continuous
integration.
Cloning
to
deploy
to
servers.
A
Individual
users
don't
impose
that
level
of
load,
but
we
set
these
limits.
We
set
them
so
high
that
almost
nobody
will
hit
them.
Since
people
can
hit
github
without
logging
in
you
can
anonymously
clone
a
public
repository.
We
actually
have
buckets
for
per
repository
per
user
ID
and
per
client
IP.
So
this
does
a
pretty
good
job
of
catching.
A
What
we
call
the
thundering
herd
problem,
which
is
where
a
whole
bunch
of
clients
arrive
and
they
want
the
same
thing
or
more
or
less
the
same
thing
at
the
same
time
and
the
load
spikes,
these
limits
start
out
high,
but
we
can
adjust
them
automatically
as
needed,
and
so
this
graph
here,
the
green,
is
the
CPU
load
on
on
a
typical
file
server.
So
you
know
it's
reasonably
healthy,
it's
about
10
and
then
a
bunch
of
traffic
comes
in
and
the
CPU
load
jumps
up
to
20.
A
It
hits
30
we're
starting
to
worry
about
impact
on
other
repositories
at
this
point
performance
of
any
other
repository
on
that
file
server,
because
we're
hosting
like
a
million
repositories
on
that
file,
server,
and
so
the
blue
here.
This
is
the
percentage
of
quotas
that
the
percentage
of
the
original
quota
that's
available
to
each
user,
so
it
starts
out
at
100
and
it
moves
basically
in
inverse
proportion
to
the
CPU
load.
A
The
second
thing
we
have
to
do:
we
have
to
predict
the
cost
of
each
operation,
so
we
need
to
know
in
advance
this
operation
that
we
are
about
to
allow.
Is
it
going
to
exceed
the
quota?
Now?
That's
it.
In
the
general
case,
it's
impossible
to
predict
the
future,
so
git
Mon
has
to
model
everything
it
it
models
per
repository
and
per
get
command.
A
A
For
that
repository
for
that
client,
IP
and
these
these
models
are
long
term
averages
so
they're,
based
on
past
behavior
and
they're,
pretty
good
at
predicting
off
at
predicting
the
future
and
then
any
operation
where
the
cost
of
that
operation
exceeds
the
available
quota
for
that
user.
That
gets
delayed.
Here's
how
that
happens.
A
Every
time
we
start
yet,
every
time
we
shell
out
to
get
it,
contacts
get
one
and
it
tells
get
Mon.
The
repository,
the
user
ID,
the
client
IP
address
and
the
name
of
the
operation
get
Mon
is
both
our
logging
system
and
our
priority
load
management
system
and
then
get
sends
the
command
schedule
and
schedule
is
just
a
question:
am
I
allowed
to
proceed
and
get
Mon
responds
immediately
with
either
continue,
which
means?
A
Yes,
it
fits
in
the
quota,
go
ahead
or
wait
and
if
get
gets
back,
wait
it
sleeps
a
few
seconds
and
tries
again
and
that's
it.
That
is
how
you
build
a
priority
system,
a
load
management
system.
That's
how
you,
how
you
prevent
get
from
tipping
over
file
servers,
is
sleep
and
then
run
schedule
again
and-
and
that's
that's
a
loop
you
sometimes
it
has
to
wait
more
than
once
and
then.
A
Finally,
when
the
get
after
the
gate
processes
out
proceed
at
the
end
of
the
process,
it
sends
how
much
CPU
time,
how
much
network
I/o,
how
much
disk
I/o
and
how
much
memory
it
used
back
to
get
Mon
to
feed
into
the
log
to
feed
into
the
model
so
that
we
will
know
next
time
in
even
better
detail.
How
long
that
that
particular
operation
on
that
particular
repository
is
likely
to
take.
So
here
it
is
in
practice.
Here's
here's!
How
git
Mon
works
in
the
real
world.
It
spreads
out
the
peaks.
A
So
let's
say
we
get
500
requests,
each
of
which
wants
to
take
5
seconds
all
at
the
same
time,
and
this
this
is
totally
typical.
We
have
customers
that
will
deploy
a
giant
repository
to
500
servers
out
in
the
cloud
somewhere
and
all
want
to
update
at
the
same
time
and
depending
on
what
they're
fetching
it
might
take
a
few
seconds
of
CPU
on
our
file
servers,
you
don't
want
to
try
to
run
2500
seconds
of
CPU
time
all
at
the
same
time,
you're
gonna
peg
the
server
for
80
seconds.
Nothing
else
will
get
done.
A
A
What
get
Mon
does
it
allows
basically,
depending
on
the
health
of
the
server
it
allows
up
to
about
40
of
those
requests
come
through
at
the
same
time
and
as
soon
as
they
finish,
another
40
come
in
and
as
soon
as
they
finish,
another
40
come
in
and
so
the
whole
thing
if
they're
cheap
they
could
fit
in
five
seconds.
If
those
requests
are
expensive,
they
could
get
spread
out
over
five
minutes
and
they're
kept
any
individual
user
is
kept
at
a
load
where
they're
not
affecting
the
rest
of
the
users
on
the
machine.
A
This
is
very
effective.
At
rate
limiting,
while
true
loops
now
I
haven't
seen
our
customers
code,
but
what
I
have
seen
is
how
it
looks
from
our
point
of
view,
and
there
are
definitely
customers
that
fetch
literally
as
fast
as
the
network
and
the
server
will
allow
knock
it
off
if
you're
doing
it
stop.
A
But
I
bet
in
a
sense
now
with
get
Mon,
it
doesn't
matter
because
what
happens
is
you
definitely
exceed
your
quota?
If
you
do
this,
and
so
some
sleeps
get
put
in
on
our
side,
if
you
forget
to
put
them
in
on
your
side
also,
you
don't
need
to
do
this
set
up
web
hooks.
But
not
everybody
knows
that
the
other
thing
get
Mon
doesn't
practice.
Is
it's
prioritizing
everyone
who's
not
exceeding
the
quota?
A
So
it's
basically
separating
all
the
traffic
on
the
machine
into
the
99%
of
people
who
aren't
going
above
their
quotas,
who
aren't
using
too
many
resources
who
are
sharing
the
machine
nicely
and
the
1%
who
aren't
and
the
1%
who
aren't
they
they
get
through
when
there
is
capacity
available,
the
most
lade
user
is
us
is
us.
We
have
a
thousand
plus
servers
and
we
deploy
a
really
big
repo
to
all
of
them
and
we
have
some
demand-side
concurrency
control.
It
doesn't
always
work
and
so
get
Mon
steps
in
and
says.
A
You're
gonna
make
the
server
too
busy
wait
and-
and
this
works
I
mean
it'sit's,
not
a
big
problem
for
your
scripts
to
get
delayed
a
few
seconds,
and
so
we're
dogfooding
this
one.
Initially
we
accepted
ourselves
from
this
from
it.
We
exempted
ourselves
from
these
quotas.
We
said
why.
Why
should
we
do
that
seems
to
work?
Fine?
The
problem
with
this
with
taking
the
thundering
herd's
with
taking
large
batches
of
requests
and
making
them
slower.
Is
it's
a
it's
a
really
bad?
A
It's
a
bad
experience
for
the
customers
whose
requests
are
being
made
slower.
It's
fine
for
the
other
99%
of
users
on
that
on
that
server,
but
it's
bad
for
the
customers
who
have
made
the
500
requests,
and
so
we
we
cache
whatever
we
can
and
I
use.
We
use
memcached
for
things
that
we
need
to
build
web
pages.
We
keep
some
longer
term
caches
on
disk
for
things
that
are
really
expensive
to
compute.
There
are
things
that
we
that
take
10
minutes
to
compute
and
so
we'll
we'll
store
them
in
the
repository
directory.
A
But
the
main
caching
thing
I'm
talking
about
today
is
a
new
system.
We've
built
called
lariat
and
the
purpose
of
lariat
is
to
solve
the
Thundering
Herd
problem.
So
getting
on
mitigates
the
Thundering
Herd
problem.
It
protects
other
users
from
the
thundering
herd
problem,
but
Larry
it
basically
solves
it.
Lariat
is
a
wrapper
around
PAC
objects.
Now
PAC
object
is
basically
the
back
end
of
every
fetch
pole
and
clone
and
lariat
wraps
it
and
says
anytime.
A
Two
or
more
customers
are
making
the
exact
same
request,
not
just
a
PAC
objects
on
a
particular
repository,
but
they
want
exactly
the
same
versions
of
exactly
the
same
files.
From
the
repository
we
now
cache
that
on
the
fly,
and
so
the
500
requests
that
come
in
from
the
from
the
deploy
servers
or
the
continuous
integration
servers.
We
can
compact
those
down
to
basically
one.
We
still
have
to
send
the
file
out
500
times,
but
we
only
do
the
CPU
work.
A
We
only
consume
the
memory
for
one
about
11%
of
our
traffic
can
be
served
from
the
Lariat
cache
now
11%
sounds
like
a
really
terrible
hit
rate.
Anybody
who
knows
anything
about
caches
like
disk
caches
and
CPU
caches
they're
in
the
high
90s,
if
you're,
if
you've
got
a
good
hit
rate,
but
with
Larry
at
11
percent
is
great
because
it's
hitting
the
11
percent
that
are
the
most
severe
performance
problems
for
the
file
servers
and-
and
here
it
is
in
practice,
I
love
this
graph
right
here.
This
is
a
graph.
A
Green
is
CPU
attributable
to
one
particular
repository
and
in
red.
Is
the
number
of
requests
per
minute
for
that
repository,
and
so
you
can
see,
we've
got
2,000
requests
a
minute.
We've
got,
we've
got
people
asking
for
30
hits
a
second
30
fetches,
a
second
for
that
repository
and
they're,
consuming
about
20
cores
of
CPU.
Our
file
servers
only
have
32,
so
if
you're
taking
20
you're
squeezing
out
a
lot
of
other
traffic,
this
is
bad,
and
this
is
with
git
Mon
in
place.
Get
Mon
is
struggling
to
keep
this
repository
under
control.
A
Anybody
want
to
guess
where
we
turned
on
there,
yet
I'm
gonna
say
right
here,
and
there
are
two
things
that
change
to
the
right
of
that
most
obviously,
the
CPU
numbers
go
from
basically
20
cores
to
one
core
and
that's
fine.
If
a
user
wants
a
deploy
to
hundreds
of
servers
takes
one
core
we're
fine
with
that.
The
other,
more
subtle
thing
that
happens
on
the
right
here
is
there
there's
more
of
the
red
so
now
they're
able
to
get
more
requests
through
more
quickly.
A
So
not
only
are
we
protecting
the
rest
of
the
serve
the
rest
of
the
users
on
the
server
we're
now
delivering
much
better
service
we're
responding
faster
to
to
the
user
who
was
causing
the
trouble
in
the
first
place
on
the
right
here.
This
graph
is
Larry
at
herd
sizes,
and
so
I
mean
it's
tough
to
read
these
the
y
axis
labels,
but
that
goes
up
to
16,000.
Sometimes
we
get
16,000
identical
requests
all
at
the
same
time.
I
don't
even
know
what
repository
that's
crazy,
but
now
we're
doing
it
for
the
cost.
A
The
CPU
cost
of
one
hit
final
thing:
I'm
gonna
talk
about
today
is
a
new
system
that
we
that
we
built
that
we're
debuting
today
publicly
called
D
get,
and
this
is
to
deal
with
failure.
So
servers
fail.
That's
reality.
If
you're
running
one
server,
one
git
server,
you
can
basically
pretend
that
it
doesn't
fail,
because
maybe
the
mean
time
to
failure
is
a
couple
of
years
and
when
it
happens,
you
have
some
downtime.
You
apologize
to
your
users,
you
go
to
the
backup
tapes
whatever.
A
When
you
have
a
handful
of
servers,
you
can
spend
a
bunch
of
extra
money
on
things
like
raid
and
redundant
power
supplies
redundant,
networking
professionally,
managed
hosting
centers,
and
you
can
kind
of
pretend
that
failure
doesn't
exist.
You
can
rely
on
things
like
raid
and
server
failover
when,
when
the
failures
happen,
but
the
more
servers
you
get,
there's
a
a
linear
or
worse
increase
in
your
in
your
individual
server
failure
rate
and
so
at
the
bottom.
This
is
a
status
tweet.
This
is
from
a
few
months
ago
it's
from
December.
This
is
what
happened.
A
This
is
what
it
looks
like
when
a
file
server
crashes
at
github.
This
was
the
server
didn't
fail
permanently
with
kernel
panic
or
something
we
we
tweet
that
we
have
an
issue
affecting
a
small
number
of
repositories.
Often
that's
github
codes
speak
for
a
file,
server
is
down
and
then
and
then
25
minutes
later
everything
is
operating
normally,
so
the
million
repositories
on
that
server
for
25
minutes.
Sorry,
that's
what
we're
fixing
here
today
with
Dee
yet
and
the
reason
it
takes
so
long
servers
take
a
long
time
to
reboot.
A
We
also
have
humans
in
the
loop
too,
to
notice
the
failure
to
manually
failover
to
the
hot
spare
for
that
server,
to
verify
that
everything's,
alright
and
then
eventually
to
tweet
that
everything's.
Ok,
our
goal
in
indie
guitar,
which
we
have
succeeded
at,
is
to
store
three
copies
of
every
repository
on
different
servers
with
very
different
failure
patterns,
so
put
them
in
different
racks,
put
them
in
different
availability
zones
and
make
sure
that
no
one
server
is
ever
solely
responsible
for
anything.
It's
ok
for
a
server
to
crash.
A
A
So
this
is
this-
is
yet
we've
overhauled
our
storage
cluster
and
to
ensure
that
every
operation
can
run
in
at
least
three
different
places
and
specifically
three
different
places
that
are
unlikely
to
fail.
At
the
same
time,
server
failures
now
have
no
user
visible
impact.
That's
a
tough
thing
to
test,
though
right
I
mean
how
do
you
unit
test
kernel,
panics,
a
Greg
said
he
can't
we
can't
either.
So
no,
we
we
test
with
sudo
reboot.
We
log
into
production
servers.
We
do
sudo.
A
Reboot
I
gave
an
internal
talk
at
github,
where
I
just
did
a
demo.
One
of
my
slides
was
way
time
switch
to
a
terminal
and
reboot
the
server
and
it
worked
and
another
good
test.
Rm
RF.
So
the
digit
has
repair
protocols.
D
get
notices
when
a
repository
has
has
stopped
working
and,
of
course,
has
two
other
copies
is
able
to
synchronize
it
using
guess
what
using
get
so
get
is
a
do.
D
get
is
distributed
yet,
but
it
is
already
distributed.
A
We're
really
just
keeping
three
standalone
get
repositories
and
using
git
to
move
file
or
to
move
objects
from
one
to
another.
Here's
how
that
looks
so
anytime,
you
push
something
now
or
or
any
other
write
that
web
edit
to
merge
whatever
it
goes
to
a
proxy,
and
that
goes
to
all
the
file
servers
for
that
particular
repository.
A
I
said
there
are
at
least
three
here's
here's
how
it
gets
to
those
three:
it
gets
to
all
three
at
the
same
time
and
the
push
or
the
write
only
succeeds
if
two
of
them
say
yes,
because
two
is
a
majority
of
three.
If
we've
got
it
on
five,
then
three
need
to
say
yes
and
so
on
when
you,
when
you've,
got
a
fetch
or
a
read
or
anything
else,
just
building
a
web
page
for
the
repository,
it
only
needs
to
go
to
one
of
the
file
servers
and
there's
really
fast
failover.
A
If
something
is
wrong
with
that
file
server,
if
you
can't
connect
to
the
file
server
via
the
connection,
refused
connection
timed
out
any
sort
of
unexpected
error,
it
just
goes
to
this
one
instead
or
it
goes
to
that
one
instead
and
there's
a
lot
of
flexibility
in
in
choosing
which
file
servers
available,
which
file
servers,
not
overloaded,
which
file
server
is
close
to
a
customer
reads,
got
a
lot
better.
When
we
started
I
started
doing
our
replication
at
the
application
level,
we
started
explicitly
knowing
which
file
servers.
A
Any
repository
was
on
with
this
rights
tolerate
single
failures,
because
I
said
we
need
two
yeses
reads:
tolerate
double
failures
view
it's
very
rare
to
get
double
failures
in
two
different
racks
or
two
different
availability
zones.
But
if
you
do
you
mark
that
server
offline
permanently
d
get
sees
that
there
only
two
copies,
it
makes
a
third
copy.
It's
all
automatic
at
this
point.
A
Here's
how
it
looks
from
the
administrators
point
of
view.
The
the
infrastructure
team
loves
D
git,
it's
failure
tolerant.
So
now
we
don't
have
to
worry
about
single
file
servers
going
down
when
we
have
to
reboot
one
for
maintenance.
We
can.
Our
old
file
servers
had
uptime
of
like
two
years,
that's
no
longer
necessary
or
useful
its
self-healing.
If
anything
is
broken
in
one
of
the
three
copies
of
a
repository,
it
fixes
it
and
it's
self
balancing.
A
So
we
want
to
make
sure
that
each
server
is
equally
full
and
each
server
is
equally
busy
and
moving
repositories
around
ensures
that
they're,
equally
full
moving,
read
load
around
ensures
that
they're
equally
busy.
This
is
the
rollout
so
as
of
this
morning,
about
60%
of
repositories
are
indeed
yet
so
I
any
of
you
who
are
github.
Customers
are
benefiting
from
this
already.
Nearly
a
hundred
percent
of
gists
are
already
indeed
yet
it
works.
If
you
haven't
noticed,
that
means
that
it
works.
A
A
They've
got
one
hundred,
they've
got
sorry,
10,000,
unique
clients,
and
normally
we
we
have
a
lot
of
repositories
with
10,000
unique
clients,
but
they
were
doing
shallow
fetches
and
anybody
who's
really
geeky
in
to
get
you
can't
reuse,
your
deltas,
so
a
shallow
fetch
instead
of
taking
one
second
of
CPU
time,
takes
30
to
60
seconds
of
CPU
time
and
there's
no
easy
way
to
optimize
that.
So
we
had
busy
servers.
The
cocoapods
users
complain
to
support.
Nobody
else
did
because
git
Mon
works.
A
C'mon
was
effectively
limiting
that
repository
to
pegging
5
CPU
cores
constantly
night
and
day
we
turned
on
detailed
logging.
That's
how
we
found
out.
They
were
doing
shallow
fetches.
We
encouraged
the
repository
owners
to
stop
doing
shallow
fetches,
but
there's
some
turnaround
time
on
that,
and
so
in
the
meantime
we
moved
this
one
to
D.
Yet
so
you
saw
the
the
rollout
ramp
goes
back
about
3
months.
We
move
cocoapods
to
D
get
suddenly
it's
got
3
servers
instead
of
1,
and
so
it's
got
three
times
the
capacity
and
we
stopped
having
to
quota
anything.
A
So
now
now
this
crazy
amount
of
load.
This
is
just
going
through
at
full
speed,
so
when
you've
got
three
servers,
it's
okay
to
dedicate
five
cores
constantly
to
them
summary
here.
So
when
you
run
the
biggest
yet
host
in
the
world,
you
see
high
traffic,
its
spiky
traffic
and
I.
Don't
just
mean
that
it's
high
after
lunch
on
the
East,
Coast
or
whatever
I
mean
we
get
deploys
where
hundreds
or
even
thousands
of
servers
asked
for
the
same
repo.
At
the
same
time,
those
are
serious
hot
spots.
A
We
get
some
corner
cases
busy
deep
repositories.
When
you
got
more
servers,
you
got
more
problems
and
failure
rate
goes
up
more
than
linearly,
so
you
have
to
write
software
to
deal
with
failure
and
our
answers.
As
I
talked
about.
We
monitor
everything.
We
log
every
single
gate
operation.
We
have
an
established
workflow
with
git
core
developers
who
can
make
it
faster
for
us.
We
cache
everything
we
introduced
Lariat.
Today
we
do
adaptive
and
predictive
load
management.
A
That's
that's
get
mons,
so
we
prioritize
light
traffic
over
heavy
traffic
effectively
and,
finally,
we
we
have
moved
replication
from
being
at
the
disk
level
and
the
server
level
to
being
at
the
application
level
so
that
we're
keeping
three
copies
of
everything
we
have
hot
spares
of
everything
and
that's
it
I.
Thank
you
for
your
attention.
I
haven't
left
much
time
for
questions,
but
I'd
love
to
catch
up
afterwards
and
you
can
find
the
blog
post
and
the
wired
article.
If
you
want
to
learn
more
about
diga.