►
Description
Presented by Zhongpeng Lin
As the size of Uber's Go monorepo grows, Git commands are getting slower and slower. We address the issue from two fronts:
(1) Sparse checkout: we developed a tool that allow users to specify a list of directories that they need to change, the tool calls bazel query to discover all files in the repository needed to build projects in the given directories and checks out the directories and their dependencies. The tool maintains a private work tree to run Bazel queries and a work tree for each sparse checkout.
(2) FS Monitor: we can share some git performance data without any fsmonitor, with Watchman, with daemon, on macOS vs on Linux for those who are not using sparse checkout
A
So,
first
a
big
background.
Uber
school
monolitho
is
a
single
git
repository
that
holds
almost
all
goco
in
Uber
and
because
we
want
to
remain
the
size
of
the
git
repository,
we
do
not
check
in
generating
code
and
Supply
libraries,
but
still
we
have
over
500
000
files
in
index
and
it's
growing
because
of
the
size
of
the
repository
we,
the
P95
status
latency
we
have
in
gold,
one
repo
is
over
20
seconds
on
Mac
and
about
three
seconds
on
the
next.
Just
imagine
you
type
your
status.
A
Why
gives
us
so
slow?
We
found
that
gift
set
has
spent
most
of
time
in
doing
two
things.
One
is
preloading
index
where
it
gives
Setters
calls
our
stats
on
each
file
and
the
second
part
is
collecting
unchecked
files
and
for
in
which
it
caused
reader
for
each
directory
and
because
of
that
I
guess
that
has
grow
linearly
as
the
number
of
fire
growth
in
the
Repository,
and
we
also
found
that
on
Linux
and
Mac
they
this
cause.
The
number
of
costs
are
very
similar.
A
So
in
order
for
gear
Setters
to
grow
to
scale
as
the
number
of
files
grow
in
the
repository,
we
have
to
break
this
linear
relationship.
There
are
two
optimization
that
we
can
do.
One
is
to
collect
file
changes
incrementally,
instead
of
like
traversing
all
the
files
at
once.
We
can
use
up
FS
monitor
to
do
the
file,
collect
file
changes
and
the
second
part
is
to
skip
all
the
directories
that
are
not
changed
since
last
time,
git
status
was
run
and
this
can
be
done
by
caching
the
time
for
all
the
directories.
A
A
However,
the
P95
latency
doesn't
improve
that
much,
because
Watchman
needs
a
very
significant
warm-up
to
like
collect
all
the
the
status
of
files.
On
the
other
hand,
on
Linux
there's
no
much
improvement
on
either
a
P95
or
medium.
In
fact,
it
make
P95
even
worse
starting
kit
version.
2.6
36.1.
We
have
a
building
FS
monitor
team
available.
So
our
our
metrics
show
that,
after
enabling
the
qfs
and
monitor
Daemon,
the
P95
latency
becomes
7.5
7.7
seconds
on
Mac
OS.
A
A
A
What
if
we
can
reduce
the
files
in
the
in
the
workspace
and
that's
the
idea
behind
guess
what?
So?
In
a
large
repository
like
Uber's
government
repo,
we
found
that
most
developers
only
work
on
a
handful
of
projects
and
most
of
the
microservices
and
libraries
that
we
have
in
Uber's
domain
repo
only
depends
on
a
small
portion
of
the
Repository.
A
So
a
simple
approach
is
to
only
check
how
sourceful,
if
you
only
work
on
the
project
food,
but
it
gets
more
complicated
than
that.
What
if
food
depends
on
another
Library
called
bar
and
what
is
bar
depending
on
other
libraries
and
what,
if
the
dependency
of
other
libraries
changes
and
the
bill,
the
build
graph
will
be
broken
and
you
won't
be
able
to
build.
A
Fortunately,
we
use
spazo
as
the
bo
system
and
bezel
require
that
every
package
has
to
explicitly
depend
declare
their
dependencies.
So
that
gives
and
give
us
an
idea.
We
can
use
spatial
query
to
discover
all
the
dependency,
the
transitive
different
instances
of
a
project
and
check
all
the
the
dependency
closure
into
git
Splash
account.
A
But
then
the
question
is
what
to
run
the
query,
because
we
need
to
run
bazel
query
to
discover
dependency
and
before
running
that
query,
the
spouse
check.
How
may
not
have
a
full
dependency
graph
in
this?
So
if
we
run
Visa
query
in
sports
check
out,
this
will
break
because
many
of
us
are
many
View
files
are
missing
and
the
other
problem
we
have
is
different.
Workspaces
may
have
different
revisions
check
out
there
and
different
revisions
may
have
different
dependency
graphs,
so
we
cannot
just
run
physically
in
arbitrary
workspace.
A
A
But
in
order
to
fold
the
pages
to
return
the
accurate
result,
we
need
to
make
sure
that
403
has
exactly
the
same
graph
as
the
sparse
checkout.
So,
but
in
order
to
do
that,
we
have
to
check
out
the
same
Revision
in
the
403
and
then
copy
all
the
uncommitted
changes
from
the
spouse
worksheet
into
the
world
tree.
A
But
after
doing
that,
the
previous
state
of
that
fork
street
is
already
gone
and
what,
if
users,
have
some
important
changes
in
that
of
work?
Three,
so
in
order
to
avoid
erasing
all
usage
changes,
we
make
the
4x3
private.
So
we
discourage
people
to
make
changes,
and
then
we
can
mutate
the
state
of
the
full
tree
at
any
time
so
pc1
together.
We,
this
is
the
typical
workflow
that
we
can
use,
use
the
splash
account.
A
A
After
that,
people
can
check
our
new
Branch
at
the
files
run,
tests
commit
do
whatever
they
need
to
develop
their
code
and
at
some
point
they
have
to
sync
with
the
main
branch
by
calling
something
like
gitpool
origin
name,
but
as
between
the
last
time,
they
check
out
the
the
spot
checkout
and
the
time
that
they
run
the
poor,
originally
some
login
library
that
they
are
depending
on.
We
have
a
new
dependency
and
then,
after
the
pool
from
me,
the
bill
may
be
broken.
A
So,
after
deploying
spot
check
how
it
we
can
see
that
it
greatly
improved
the
git
status
performance
on
both
Linux
and
mac
and
the
P95
on
both
systems
are
close
to
once
again
or
sometimes
even
less.
And
this
also
summarize
all
the
optimization
that
we
have
done
on
Uber
scope
on
a
repo
in
the
future.
We
can
deploy
a
video
query
services
that
running
online
so
that
we
don't
need
to
maintain
a
full
of
three
on
every
laptop
and
we
are
also
thinking
about
deploying
partial
grounds.
B
A
Yeah,
it's
some
something
that
we
definitely
want
to
explore
right
now.
We
we,
we
are
more
focusing
on
index
to
provide
the
difference
between
sponsorship
versus
checkout,
but
is
the
Smart
index
can
be
deployed
by
integrating
into
a
git
the
kpcr
tool?
That's
internally
manage
the
whole
git
repository.
Yes,
it's
telling
something
there
in
our
radar.
A
Yeah
good
question:
so
in
Uber
we
have
something
called
local
developer
analytics.
So
we
have
a
good
wrapper
that
is
they're,
sending
development
Matrix
to
our
our
servers.
And
then
we
know
the
we
collect
almost
every
git
command,
the
the
latency,
the
version
or
even
the
username
of
the
people
of
the
their
laptop
yeah.
So
we
have
we
get
all
those
data
so
that
it
allows
us
to
try
all
different
configuration
and
see
how
it
affects
the
performance
of
developer
experience,
developer
laptops
in
a
large
scale.
A
D
I
guess
it's
a
little
bit
off
topic,
but
you
mentioned
that
you're
Gathering
metrics,
using
this
wrapper
around
kit.
Have
you
looked
into
using
the
gets
Trace
2
library
to
to
gather
metrics
from
traces?
That
way,
is
there
something
like?
Did
it
not
meet
your
needs
in
some
way?
Is
there
a
reason
that
you're
using
a
wrapper
instead.