►
Description
Presented by Ivan Frade, Minh Thai, Software Engineers, Google
"Google runs a Git hosting service that handles 100s of thousands of repositories of all shapes and sizes. We host Android, which contains huge binary assets and employs over 1000 distinct repositories, and Chromium, a monorepo with more than 1750 unique committers last year. Both projects see high volumes of code review traffic and employ significant automation (build bots, auto mergers). We see bursts of greater than 20 updates/second in individual repositories, and a bug fix in one branch can trigger a cascade of automated updates to other branches and repositories.
This talk will provide tales from the trenches of seeing Git stretched to new limits, the work that has motivated, and how Google monitors its hosting service to ensure Git can continue to scale."
About GitMerge
Git Merge is the pre-eminent Git-focused conference: a full-day offering technical content and user case studies, plus a day of workshops for Git users of all levels. Git Merge is dedicated to amplifying new voices in the Git community and to showcasing the most thought-provoking projects from contributors, maintainers and community managers around the world. Find out more at git-merge.com
A
So
a
lot
of
people
ask
me
like
what
is
the
interest
of
Google
on
on
it,
and
actually
we
have
quite
a
lot
of
things
to
do
in
our
team
because
we're
the
team
that
hosts
the
Android
repository
Android
is
probably
the
biggest
open
source
project
out
there
with
a
lot
of
active
development.
So
a
huge
volume
of
commits
coming
and
going
is
also
a
repository
that
contains
large
binary
assets,
which
is
always
a
pain
point.
Indeed.
A
We
also
host
chromium
if
Android
is
the
biggest
repository
of
the
chromium
lycée,
also
on
the
top
of
on
sides
of
projects
with
the
extract
list
that
it
runs
as
a
single
mono
repo,
then
we
are
also
the
service
that
hosts
the
code
of
other
alphabet
companies
like
nestin,
very
ly,
and
finally,
we
also
run
the
software
that
powers
the
cloud
source
repository,
one
of
our
offerings
in
the
Google
Cloud,
where
well,
is
our
hosting
service
where
users
can
create
repositories.
So
this
puts
us
in
the
order
of
millions
in
repositories
in
total.
A
So,
what's
the
problem
with
this,
that
we
are
seeing
repositories
with
the
order
of
of
10
million
objects
and
what's
the
problem
with
that
word,
usually,
when
you
run
a
git
clone
on
your
repository,
you
usually
get
a
stuck
in
that
fourth
line,
the
one
that
says
that
we
are
downloading
the
content.
You
are
there
wishing
that
your
internet
would
be
faster
and
you
barely
notice
that
there
is
this
second
line.
That
says,
counting
objects
that
you
sell
is
very
fast
in
most
of
the
repositories.
A
A
Analyzers,
then,
if
you
run
your
code
review
systems,
some,
like
the
rate,
are
based
on
branches
and
creating
even
more
references-
and
this
is
a
problem
for
that
is
well
explained
in
this
paragraph
from
the
from
the
git
protocol
spec,
and
it
says
that
when
you
run
a
fetch
when
the
client
initially
connects,
the
server
will
immediately
respond
with
a
listing
of
each
reference.
It
has
this
works.
A
Well,
when
you
have
a
few
when
you
have
a
lot,
but
when
you
have
in
the
head
of
the
million,
this
doesn't
scale,
we
start
sending
to
the
client
megabytes
of
information
that
probably
the
client
does
having
even
care
about
the
poor
guy.
Just
wanted
master
so
and
also
on
the
server
side
depends.
How
do
you
structure
your
back-end,
and
how
do
you
store
your
information?
Well
retrieving.
A
All
those
references
is
not
a
trivial
task
and
then
an
order
of
the
promise
that
you
get
with
the
scale
is
that
not
everybody
uses
a
geat
as
the
standard
repository
of
your
source
code,
you
build
or
commits
you
keep
your
development
history
and
you
share
your
changes.
So
some
people
comes
to
gate
with
inherited
workflows,
maybe
porting
from
a
different
version
control
system.
Then
they
try
to
emulate
some
of
those
features
or
workflows
and
sometimes
not
everything
fits
perfectly
with
get.
A
A
A
A
Well,
the
five
thousand
references
of
the
kernel
are
are
not
that
much
if
you
compare
with
the
two
million
that
we
get
so
in
Android,
ya,
think
on
those
two
million
that
is
tax
or
on
branches.
If
you
need
our
references,
if
you
need
to
send
them
back
every
time,
you
try
to
do
a
fetch
and
then
the
kernel
has
seven
million
objects.
Well,
chromium
has
more
than
20
million,
and
when
you
put
that
all
together
in
your
pack
file,
so
you
end
up
with
a
with
a
repository
of
23
gigabytes.
A
A
So,
on
the
conversation,
Model
S
goes
like
this.
The
client
says
hi
the
server
answers.
Yes,
look
I
have
master
pointing
to
this
committee
gee
and
maybe
other
references.
The
claim
says:
oh
I,
don't
know
about
that.
So
I
want
to
that
commit
G,
and
the
last
thing
that
I
have
is
C,
so
figure
out
what
is
happening
here.
A
Once
you
start
having
branches
merge
commits
and
a
more
complicated
topology
things
are
not
that
easy,
and
also
what
happen
is
that
to
do
that,
you
need
to
read
the
commit
parse.
It
extract
the
parent
check
if
it
is
in
your
half,
if
not
repeat
this
process,
so
actually
retrieving
a
lot
of
information
to
serve
all
this.
We
came
up
with
the
idea
of
the
reachability
map
bitmap,
the
rhetorically
bitmap
is
a
small
bit
structure
that
is
attached
to
the
commit
and
tells
you
what
other
comets
are
reachable
from
it.
A
So
assuming
a
tree
like
you
see
on
the
left,
so
again
how
git
stores
the
information
is,
putting
all
the
obvious
one
after
the
other
in
what
is
called
a
pack
file,
and
then
it
has
a
pack
file
index
that
says
this
commit
is
in
this.
A
opiate
ID
is
in
this
offset.
This
commit
ID
is
in
this
offset
and
so
on.
So
you
end
up
with
a
list
of
comets
and
the
bitmap
for
a
commit.
A
Pretty
much
has
a
zero
if
it
can
reach
that
other
commit
or
a
1
a
0
if
it
cannot
reach
it
or
a
1,
if
it
can
so
in
this
case,
for
example,
from
from
the
node
G,
if
you
would
work
backwards
on
the
tree,
you
can
reach
every
other
node
except
B,
so
it
has
a
1
in
every
position,
except
on
D
that
it
has
a
0
if
you
check
the
other
extreme.
For
example,
in
the
case
of
a
it,
has
a
0,
you
cannot
reach
anything
from
node
a.
A
The
nesting
that
we
saw
is
that
it
is
designed
for
a
distributed
development
environment
where
you
have
just
developers
with
their
local
copy,
doing
their
work
and
sharing
a
small
changes
and
the
clone
is
pretty
much
a
one-time
operation
very
expensive,
but
you
just
do
it
at
the
beginning
and
that's
it.
But
what
we
are
saying
is
that
there
are
other
cases
and
and
Wester
seen
cases
like
the
containerized
continuous
integration,
for
example.
This
is
a
short-lived
client.
A
It
just
want
to
come
up
grab
a
copy
of
the
code,
probably
of
the
NASS
commit,
or
maybe
the
last
two
commits
to
make
some
analysis
on
calculating
dibs
and
do
their
work
and
disappear,
and
definitely
the
default
clone
is
terribly
expensive
for
this
case.
So
then
we
have
the
option
of
the
shallow
clone.
A
Shallow
clone
is
I
think
the
conceptual
is
very
easy.
So
if
you
have
a
tree
like
this,
with
the
commits
a
B,
C
and
D,
each
one
has
its
tree.
That
inside
can
have
another
blob,
some
files
and
maybe
other
nested
trees
and
all
this
structure
spawning.
So
when
the
client
says
I
want
just
a
shallow
copy
of
this
with
a
depth
of
2,
it
will
receive
the
just
the
commit
D
commit
C
and
everything
hanging
from
there.
A
A
The
optimizations
that
we
talked
before
with
the
bitmap
are
for
commits
not
for
the
whole
trees,
so
we
still
need
to
dig
through
the
through
the
PAC
file
to
tract
all
those
trees
and
objects
that
are
needed
so
far.
We
don't
have
a
very
good
solution
for
that
blues
thing,
that
if
you
have
a
pack
file
like
chromium
of
20
gigabytes,
well,
good
luck
of
trying
to
get
pieces
of
that
brand.
Only
and
still
shallow
plan
seems
to
be
a
good
solution
for
this.
Don't
confuse
it
with
another
feature
that
is
coming
knowing
it.
A
That
is
the
partial
clone,
because
in
the
case
of
the
partial
clone,
the
client
actually
knows
all
the
obvious.
In
the
case
of
the
shallow
everything
below
below
that
cut
point
is
simply
invisible
and
the
partial
clone
we
just
know
it
the
Aria
second
to
the
server,
don't
send
it
to
me
now.
I
will
tell
you
later.
A
B
You
good
morning
merge
I'm
gonna
switch
gear
a
little
bit
and
move
away
from
the
theoretical
implementation
that
eval
have.
You
know,
went
through
and
show
you
some
example
of
the
use
cases
that
keep
us
as
a
get
holster
beyond
our
toe
so
see
most
version
control
system
outside
get
have
some
notion
of
revision.
Number
that
is
increasing.
It
may
be
continuous
or
or
non
continuous,
but
it's
it's
always.
You
know
monotonic
and
and
that's
very
useful,
because
it
kind
of
give
you
a
sense
of
directions
of
okay
I'm
going
that
direction.
B
As
a
matter
of
fact,
a
lot
of
organization
will
celebrate
if
the
developer
have
checked
in
some
milestone:
number
of
changes,
I
out,
one
thousand,
ten
thousand
or
million.
So
it's
it's
part
of
the
culture
of
software
development
and
you
know,
get
comes
around
and
I'm
see,
get
use,
has
as
a
revision
number
and
has
it's
not
increasing
and
for
a
log
organization
that
would
be
a
blocker
from
in
deploying
get
as
a
revision
control
system.
So
software
developer,
you
come
up
with
solution.
B
So
here's
the
example
that
we
have
with
chromium
the
chromium
project
moved
to
get
in
2009
almost
ten
years
ago
and
they
migrated
from
SVN
and
if
you're
familiar
SVN
have
increasing
revision.
Number
that
you
know
would
send
back
to
you
after
you
do.
The
commit
and
a
lot
of
the
software
has
been
built
to
rely
on
this
property
and,
like
I,
said
before,
the
developers
often
refer
to
each
to
the
vision
number
in
communication
between
organization
and
that's
nice.
B
B
After
for
every
commitment
to
this
branch,
they
would
have
a
bar
that
watches
brands
and
copy
that
commit
over
to
the
main
master
branch,
and,
while
doing
so,
it
alter
the
commit
message
to
attach
a
revision
number.
So
this
worked.
Actually,
it
works
for
many
years,
but
as
a
side
effect
that,
as
tango,
buys
and
revision.
B
B
The
the
the
lessons
from
this
this
experience
is
we
not
try
to
make
a
number
into
the
commit,
because,
if
it's
subject
to
change,
you're
gonna
get
into
trouble,
because
when
you
rewrite
history
using
rebates
or
something
things,
gonna
get
mismatched
and
potentially
out
of
order
really
quickly.
So
so
the
question
you
may
have
is
what
happened
to
the
chromium
projects.
B
Yes,
it's
a
it's
go
against
our
recommendation
earlier,
but
they
they
they
want
what
they
want,
so
they
got
what
they
want.
Okay,
so
the
second
story
is
a
lot
more
recent
and
it's
not
necessarily
about
something
that
the
client
do
that
against
our
recommendation.
So,
which
is
ironic
Garrett
is
a
coach
code
review
system
that
built
on
natively
on
top
of
jacott,
which
is
a
Java
implementation
at
the
gate
protocol
and
here
in
the
graph,
is
a
typical
commit
tree
of
a
you
know,
garrett
project.
So
you
got.
B
You
got
two
changes
going
on
change,
one
on
the
left
and
change
two
on
the
right
chain.
Ones
on
the
left,
have
two
different
paths
set
and
change
to
on
the
right.
Have
three
paths
set,
so
you
got,
you
know:
lots
a
bunch
of
refs
pointing
into
a
bunch
of
branches
everywhere
and
gets
over
loves
this,
because
you
got
a
big
tree.
We
can
build
a
bitmap,
no
problems,
but
recent
development
in
the
garret
did
a
great
system.
B
B
It's
behavior
to
the
data
store
legs
in
a
datastore,
which
is
quite
novel
because
you
have
one
single
git
repository
that
contain
all
the
history
of
the
KO
chains
and
review
that
makes
this
coach
chains
approve,
which
is
very
nice,
but
the
problem
is
the
or
the
chains
meta
is
store
in
in
a
separate
tree
than
the
country
and
there's
no
relation
whatsoever
between
the
node,
which
is
denoted
in
red
in
this
graph
and
anytime,
for
every
single
path
set.
There's
a
new
branch
that
created
for
that.
B
So
you
end
up
having
a
lot
and
lots
of
little
little
commits
to
sort
through
whenever
due
to
negotiations
and
as
I
mention
before
a
lots
of
commits
is
bad
for
negotiations
and
and
this
is
becoming
a
real
problem
as
attempt
go
by
and
you
have
thousands
of
active
change
happening
so
that
the
change
that
you
have
is
actually
more
recent
than
the
change
in
the
master
branch.
So
negotiation
would
give
up
because
it
doesn't
want
to
take
so
long
before
it
finishes.
B
B
There's
protocol
v2,
which
is
the
biggest
Saviour
for
us,
which
rafting
one
we
can.
The
client
can
specify
the
list
of
referent,
so
the
server
can
say:
oh
if
we're
gonna,
just
interested
in
this
raft,
who's
just
gonna,
negotiate
within
the
commit
from
this
graph
and
that
go
from
these
hundreds
of
thousands
of
raft
into
just
a
couple
and
that
helps
a
lot.
B
Another
chain
that
was
done
in
the
proko
is
the
ability
to
make
this
negotiation
round
bigger.
So,
instead
of
sending
you
know
30
to
Sarah
one
in
one
round,
you
said
you
know
256
something
because
the
each
of
the
negotiation
route
context
time
for
round
trip
and
another
change.
That
is,
we
go
longer,
so
you
trade,
the
time
negotiating
for
the
time
of
downloading
another
chain
in
the
protocol,
was
very
useful,
was
instead
of
walking
back
in
the
commit
graph
one
by
one.
It
can
actually
skip
in
some
fashion.
One
other
way
to
do.