►
Description
Microsoft is in the middle of moving the majority of its own product development onto Git. Yes, that Microsoft. From small codebase to large. Over 15k developers are already using Git today, with 10’s of thousands yet to come. The journey to get there has been an interesting one, full of ups and downs. Saeed will discuss Microsoft's journey and the challenges along the way, both social and technical.
Saeed is on the Visual Studio Team Services team at Microsoft, focused on helping some very large teams within Microsoft migrate to Git. Among other things, this means solving some hard scale problems in Git, which is a great source of fun. He also enjoys rock climbing, road biking, and music.
About Git Merge:
Git Merge is the pre-eminent Git-focused conference: a full-day offering technical content, user case studies and training workshops for Git users of all levels.
Visit: http://git-merge.com/
A
The
IMF
p.m.
on
visual
studio
team
services
I
own
our
get
serviced,
we
ship
that
you
know
for
external
customers,
but
we
also
use
it
heavily
inside
the
company.
That's
what
I'm
going
to
talk
about
and
I'm
a
much
less
skilled
presenter
than
all
the
people
have
gone
before
me.
So
if
you
look
real
carefully,
it
I
think
it's
on
slide.
Eight
you'll
see
some
color,
but
I'm
gonna.
Try
to
talk
about
our
process
is
a
company
of
moving
towards
adopting
one
unified
engineering
system.
A
So
you
know
the
old
Microsoft
Way
was
every
team
kind
of
did
their
own
thing.
We
had
a
really
heterogeneous
environment.
Any
team
of
any
considerable
size
would
have
to
decide
what
version
control
system
they
want
to
use.
They'd
have
to
build
their
own
build
system
and
test
frameworks
and
build
their
own
workflows
around.
All
of
that
and
the
there
wasn't
a
lot
of
sharing
across
these
team
boundaries
and,
of
course,
that
kind
of
environment
creates
all
of
the
problems
that
you'd
you'd
expect.
A
A
If
you
can
find
the
right
team
that
will
talk
to
to
that,
and
you
need
them
to
make
a
change
in
a
piece
of
code
that
you
need.
It's
often
really
hard
to
convinced
somebody
to
go.
Do
work
on
your
behalf.
You
know
it's
a
lot
easier
to
just
give
them
code.
That
already
does
it,
but
you
can't
do
that
if
you
can't
even
find
the
code,
it
also
just
deters
people
from
from
moving
around,
because
every
time
you
move
to
a
new
company,
not
a
new
team.
A
You
have
to
learn
a
whole
new
engineering
system,
so
we
figured
there's
got
to
be
a
better
way
and
we're
trying
to
move
towards
a
unified
engineering
system
for
the
whole
company.
One,
that's
centered
around
common
tools,
common
workflows,
we
use
get
heavily.
We
get
heavily
worth
building
a
common,
build
and
test
infrastructure,
but
really
the
most
important
piece
of
all
of
that
is
to
have
a
culture
of
sharing
code
across
the
company.
You
know:
we've
we've
been
heavy
contributors
to
open
source
and
we
have
begun
to.
A
You
know,
use
open
source
quite
heavily
as
well,
but
one
another
thing
that
we're
taking
from
that
is
the
culture
of
being
open
sharing
code
and
and
encouraging
people
to
work
on
contributing
back
rather
than
having
their
own
their
own
territory,
so
to
speak.
So
the
culture
and
the
tooling
really
go
hand
in
hand.
You
have
to
have
the
right
tools
to
encourage
the
right
culture,
but
you
have
to
have
the
culture
in
order
to
use
the
tools.
A
So
you
know
we
started
off
with
this
vision
a
couple
years
ago.
You
know
imagine
a
world
where
and
all
these
these
things
you
know
imagine
where
you
can
find
anybody
can
find
the
source
code.
Anybody
can
make
changes.
All
the
intellectual
property
of
the
company
is
open
to
everyone
in
reusable
components.
These
are
all
things
that
are
not
revolutionising
in
the
world
we
live
in
now,
but
it
sure
are
quite
different
from
where
things
were
20
years
ago,
or
even
ten
or
five
years
ago.
A
Of
course,
the
the
core
of
an
engineering
system
is
version
control,
and
we
think
that
git
is
a
good
foundation
for
our
unified
engineering
system.
You
know
it
get
brings
with
it
both
the
tooling
characteristics
and
workflows
that
enable
sharing,
but
also
it
brings
with
it.
The
culture
of
sharing
so
much
of
the
open-source
world
is
in,
is
then
get,
and
you
know
they
just
fit
really
well
together.
So
across
the
company
we've
we've
already
been
adopting
it
quite
heavily.
A
Many
teams
are
already
fully
on
get
Visual,
Studio
is
in
get
our
team
Visual
Studio,
Team
Services
is
fully
and
get
Azure
and
Skype,
but
there's
also
dozens
of
smaller
teams.
Hundreds
of
personal,
probably
thousands
of
personal
repos
and
many
many
more,
are
in
the
process
of
migrating
migration
brings
a
lot
of
challenges
with
it.
I
think
you
know,
we
had
a
talk
earlier
today
about
training
people
to
understand,
get.
That
is
probably
one
of
the
biggest
bottlenecks
of
people
who
have
been
working
for
years
and
centralized
source
control.
A
You
can't
just
throw
them
in
a
new
repo
and
say
go
with
a
little
bit
of
a
tutorial.
You
actually
have
to
retrain
all
the
people.
You
have
to
build
new
workflows,
you
have
to
rebuild
your
tools
and
get
it
make
sure
everyone
can
be
productive.
So
a
lot
of
teams
are
in
the
process
of
migrating
for
the
teams.
That
already
are
we've
begun
to
achieve
a
lot
of
the
the
goals
that
we
set
out
to
achieve.
The
vast
majority
of
our
repos
are
visible
to
anyone
in
the
company.
A
The
default
of
any
repo
that
we
create
is
that
it's
open
anyone
in
the
company.
So
not
only
is
it
open,
but
it's
in
a
shared
place
that
anybody
can
go
find.
You
can
do
a
code
search
across
a
lot
of
these
repos
and
find
you
know
code
that
you're
looking
for
more
importantly,
anyone
can
submit
a
pull
request
inside
any
of
these
repos
and
so
we're
already
building
this
culture
of.
If
you
need
something
you
don't
show
up
with
a
request,
you
show
up
with
a
portal,
you
don't
show
up
with
a
verbal
request.
A
You
show
up
with
a
pull
request
and
the
the
build
and
test
infrastructure
is
still
are
not
fully
standardized,
but
pull
requests
really
do
go
a
long
way
to
enforce
the
quality
bars
that
any
team
wants
to
set
so
say
for
anyone
to
come
in
and
play
you're
not
gonna,
be
able
to
break
master
if
they've
configured
it
that
you
can't
push
the
master,
you
have
to
submit
a
full
request,
so
the
tools
are
really
helpful.
There
you'll
see
this
is
our
hey
snow
color.
A
This
is
our
get
usage
across
the
company
over
the
last
several
quarters,
and
just
so
we're
specific
on
what
these
numbers
mean.
These
are
people
that
have
done
multiple
interactions
with
our
git
service
over
the
course
of
a
month.
So
they've
done
a
fetch.
They're
gonna
push
they've
created
a
pull
request.
They've
reviewed
a
pull
request,
so
they've
done
multiple
meaningful
interactions
with
our
git
service
over
the
course
of
the
month.
In
order
to
show
up
on
this
graph
and
so
usage
is,
is
building
quite
rapidly.
It's
almost
a
hockey
stick,
maybe
not
quite
I.
A
A
So
discoverability
is
an
issue
creation
of
those
repos
and
permission
models
around
them
and
managing
all
the
policies
on
them
risk.
You
know,
there's
a
lot
of
Management
scaling
issues
there
that
we're
working
on
solving.
We
also
have
a
need
to
go
quite
large
in
some
repos
there's
teams
that
just
have
lots
of
people
or
lots
of
content,
and
so
we
have
to
be
able
to
scale
up
to
some
quite
large
numbers
all
problems
that
we're
working
on.
But
for
now
I
want
to
talk
to
you
too,
about
our
own
visual
studio
team
services
team.
A
We
have
what
I
consider
a
medium-sized
team
and
and
repo
300,
or
so
the
number
varies,
but
about
350
or
so
I
think
engineers
working
in
the
code
and
actively
writing
code.
Our
repo
is
about
2
gigabytes
of
sources
that
tip
and
that's
actual
sources.
It's
not
there's
no
binaries
checked
in
about
a
hundred
thousand
files,
and
our
cadence
is
that
every
three
weeks
we
ship
a
new
iteration
of
the
service
we
deploy
our
service,
but
also
every
quarter.
A
We
should
have
a
box
product
team
foundation,
server
and
those
are
coming
out
of
the
same
codebase
and
they
have
different
cane
release
Cadence's
with
different
requirements
around
each,
and
so
we
made
an
initial
migration
to
get
a
couple
years
ago,
I
think
two
and
a
half
or
so
years
ago.
At
this
point
where
we
we
took,
you
know,
team
foundation,
server
and
visual
studio.
A
Team
services
are
comprised
of
a
bunch
of
different
services
that
work
together,
and
so
our
initial
approach
was:
why
don't
we
take
each
of
those
services
put
them
in
internal
and
repo?
We
can
build
them
separately,
each
of
them
will
be
smaller
and
we
built
some
additional
tooling.
Like
I
said
we
every
once
a
quarter.
We
want
to
bring
all
these
together
and
make
sure
that
they're
all
compatible
with
each
other,
and
all
you
know
that
all
their
dependencies
are
the
same
version,
so
we
have
to
be
able
to
build
them
all
together.
A
So
we
built
some
tooling
that
stitched
all
these
repos
together,
but
really
it
was
a.
It
was
a
big
pain
for
us.
The
the
code
boundaries
between
those
services
are
not
as
clean
as
we
would
like,
because
in
the
past
it
was
all
they
were
all
in
one
centralized
version
control
system
and
there
wasn't
a
need
for
necessarily
super
clean
boundaries,
and
so
we
just
hadn't
done
that
work
yet
and
the
the
versioning
problems
that
we
had
when
we
wanted
to
ship
the
box
products.
A
Every
three
months
caused
a
lot
of
pain,
and
so
it
just
the
result
of
this.
Of
all
this
was
that
developers
had
a
lot
of
pain
working
across
these
repo
boundaries.
We
had
a
lot
of
versioning
that
we
had
very
low
engineering
velocity,
and
so
we
took
us
look
at
what
we
had
done
and
try
to
find
a
better
way.
You
know,
after
this
we
looked
at
sub
modules
and
we
actually
built
some
rather
robust,
tooling,
around
sub
modules,
the
problem
with
sub
modules.
A
They
work
just
great
for
the
problem
that
they
solve
and
you
can
build
tooling
to
solve
the
problem
that
we
had,
but
it
compounds
that
education
problem
and
not
only
is
get
just
a
dag
or
it's
not.
But
when
you
sub
modules,
now
you
have
a
collection
of
loosely
connected
tags
and
the
mental
overhead
is
just
exponentially
higher
and
so
yeah.
We
built
the
tool
and
it
works,
but
we
realized
that
we
were
gonna
set
our
developers
up
for
even
more
pain.
A
So
you
know
the
really
that
what
we
learned
from
all
that
was,
you
cannot
make
your
repo
granularity
be
smaller
than
your
build
granularity,
and
any
attempt
to
do
so
is
just
causes
of
pain
for
your
developers.
So
there's
there's
two
paths
forward
to
solving
this.
You
have
to
componentize
your
code
better.
If
you're
coming
from
a
monolithic
code
base
and
a
larger
team,
you
want
it,
you
have
to
componentize
your
code,
but
that's
not
a
quick
and
easy
process.
It's
a
no
often
a
heavy
investment.
A
It
takes
a
long
time
and
you
also
need
to
be
able
to
support,
get
repos
that
are
quite
a
lot
larger
in
some
cases
then
than
we
had
anticipated
so
we're
working
on
both
of
those
problems.
You
know
we,
since
we
are
probably
one
of
our
products
that
we
build,
is
to
get
service,
we're
in
a
unique
position
to
make
our
own
lives
better
by
improving
those
tools.
So
what
we
did
about
a
year
and
a
half
ago,
is
we
moved
all
that
code
into
one
repo
and
some
smaller
services
as
they
do?
A
In
parallel
with
that,
we
had
to
invest
in
our
build.
Well,
you
simply
cannot
put
that
many
people
and
that
much
code
into
one
repo
without
investing
in
how
you
validate
the
code
and
I
say
build,
really
is
a
proxy
for
validating
the
changes
that
a
developer
is
submitting.
You
have
to
be
able
to
build
it
run
your
unit
tests
do
any
sort
of
validation
that
you
need
to
do
to
make
sure
that
the
changes
are
good.
So
we
invested
really
heavily
in
our
build
process.
A
We
totally
cleaned
up
our
all
of
our
build
scripts
and
full
builds,
went
from
45
minutes
down
at
less
than
10
minutes.
We
in
the
past
couldn't
doing
reliable,
incremental
builds,
and
now
we
can
and
the
net
benefit
of
having
a
really
fast
and
really
fast
is
relative
right.
10
minutes
is
still
longer
than
than
I
would
like,
but
it's
way
faster
than
where
we
were
the
benefit
of
that
is.
You
can
have
a
much
shallower,
branching
structure.
A
You
know
the
the
model
that
we
like
to
use
in
our
team
is
anybody
can
submit
changes
to
the
master
branch,
but
they
have
to
pass
the
quality
bar
and
that
simply
doesn't
scale
if
your
quality
bar
takes
an
hour
to
run,
but
it
can
scale
if
you
take
less
so
it's
really
worthwhile
to
invest
in
in
those
things.
So
now
where
we're
at
is,
we
have
one
large
repo.
All
of
our
development
happens
in
topic
branches
that
people
spin
off
of
the
master
branch,
so
master
is
just
always
trudging
along.
A
A
For
as
long
as
any,
and
then
they
submit
a
pull
request
to
submit
to
merge
those
changes
into
master
once
the
Sprint,
we
create
a
release
branch
off
a
master,
because
that's
the
that's
the
build
that
we're
going
to
deploy
to
the
service,
and
if
we
need
to
make
hot
fixes
again,
we
create
topic
branches
off
of
that
release,
branch
and
pull
request
them
back
in
and
actually
developers
can't
do
it
any
other
way
our
service.
Won't.
A
We
configure
it
through
permissions
and
branch
policies
so
that
you
cannot
push
to
master
or
to
a
release
branch
you
have
to
the
only
way
to
get
a
change
into.
One
of
those
branches
is
to
use
a
pull
request
and
our
pull
requests
are
then
configured
to
run
the
build
required
code
reviews
run
the
unit
tests,
and
you
know
so.
We
always
have
good
quality
in
our
master
and
release
branches.
The
other
thing
that
the
pull
requests
do
for
us
is
they
solve
a
scale
problem.
A
Anyone
who
has
worked
in
a
larger
team
and
has
tried
to
push
to
a
branch
like
like
master
or
any
sort
of
a
shared
branch
knows
that
it
it
can
be
really
painful.
You
know
you,
you
have
to
fetch
the
latest.
You
merge
your
change,
your
topic
branch
into
master,
and
then
you
try
to
push
now.
You
can't
because
somebody
else
beat
you
to
it.
So
you
have
to
fetch
again
merge
again
and
push
you
still
can't
cuz.
Somebody
else
beat
you
to
it.
You
know.
A
So
this
is
a
very
simplified
view
of
our
branching
structure.
The
black
line
is,
is
master
and
you
know
topic
branch
can
spin
off
a
master
and
get
merged
back
in
we
have
sometimes
you
just
need
a
longer
live
branch
because
you
need
two
or
three
people
working
on
something
together,
so
we
create
feature
branches
off
a
master
where
a
few
people
can
collaborate,
they
can
each
create
topic
branches
off
of
that
merge.
A
Those
results
together
and,
finally
do
a
pull
request
of
that
feature
back
into
master
and
at
the
top
there
I
got
a
quick
little
release
branch
spinning
off
so
and
we
spin
those
off
once
every
three
weeks
like
I,
said
this
process
works
really
well
for
us,
and
you
know
one
one
thing
we
learn
from
this:
is
you
have
to
have
when
you
have
that
many
people
in
a
branch
you
have
to
have
a
very
consistent
naming
pattern
for
your
branches.
So
what
we
do
is
we
use
this?
A
This
pattern
of
type
slash,
the
name
right
so
I
have
releases.
M98
was
a
branch
that
was
spun
off
for
the
milestone
number
98.
If
I
want
to
make
a
small
fix
in
a
topic,
branch
I
would
name
it
user
slash
my
alias
slash
whatever
name
I
want,
and
this
helps
people
organize
the
branches
it
helps.
People
discover
what
branches
they
need
to
find.
You
know
when
they're
looking
for
them,
of
course,
there's
a
lot
of
questions
around
like
you
know
what
are
best
practices
about.
They
have
short-lived
short-lived
branches
or
long
live
ranches.
A
Do
your
merger
rebase?
The
answer
to
all
these
things
is:
never
one
there's
number
one
answer
for
the
question
right.
It
really
depends
on
the
situation
so
in
arc.
In
our
case,
you
know,
master
and
releases
branches
are
always
long-lived
the
features
and
user
branches
or
a
short-lived,
and
they
have
different
policies
that
govern
how
they're
handled
you
know.
We
we
configure
the
master
and
release
release
branches
so
that
you
cannot
force
push
to
them.
You
can't
delete
them
unless
you
have
special
permission,
but
most
people
don't
you
cannot
push
to
them
at
all.
A
Unless
you
use
a
pull
request
where,
as
user
branches,
you
can
do
whatever
you
want,
you
can
create
them.
However,
you
like
you,
can
delete
them
whenever
you
like,
you
can
rebase
them,
because
then
you
can
force
push
them.
You
know
and
that's
okay,
so
our
stance
on
merge
versus
rebase
is
no.
You
absolutely
cannot
rebase
an
important
branch
right
and
sure
you
can
go
ahead
and
do
whatever
you
locally,
but
you
can't
push
it
to
the
server.
A
So
you
can't
inflict
that
on
anyone
else,
but
with
a
topic
branch
we
actually
encourage
people
to
rebase,
because
that
creates
a
cleaner
history
with
fewer
lanes
in
it
and
if
you
have
300
people
merging
into
master
and
they
never
rebase.
You
end
up
with
these
really
wide
graphs
and
so
any
any
tools
that
try
to
show
you
the
graph,
actually
become
really
hard
to
use.
Because
you
can't
you
can't
find
your
way
through
all
the
noise
by
encouraging
people
to
rebase
on
to
the
tip
of
master
before
they
merge
in
and
by
the
way.
A
Our
pull
requests
always
do
a
no
fast-forward
merge,
so
you
can't
do
a
rebase
and
then
it
fast
forward.
You
have
to
do
a
rebase
and
then
a
pull
request
which
always
creates
a
merge
commit.
So
you
always
end
up
with
a
clean
graph
that
shows
you
the
the
path
of
master
you
know,
and
if
you
look
at
the
first
parent
history
of
master,
you
always
see
the
exact
history
of
that
branch
and
all
the
branches.
A
All
those
merge
commits
that
went
into
it,
but
you
can
look
at
the
the
other
side
of
the
merge
commit
the
the
second
parent
to
see
what
the
topic
branch
was,
and
it
creates
a
really
clean
history.
Of
course,
you
know
users,
don't
remember
all
these
rules.
The
the
way
to
make
people
successful
is
to
enforce
them
on
the
server,
and
so
we
we
set
up
all
the
permissions
and
the
branch
policies
to
make
this
happen.
A
The
end
result
for
us
is
that
we
have
improved
velocity
and
improved
code
quality
and
because
we're
a
largest
team
on
the
server,
we
feel
the
pain
of
these
sorts
of
things,
and
so
we're
very
motivated
to
fix
them,
which
makes
life
better
for
all
of
our
customers
as
well,
and
one
huge
benefit
of
this
for
us
has
been
like
we're
actually
achieving
what
we
wanted.
We
get
significant
contributions
from
others
in
the
company
nowadays,
when
somebody
comes
to
me
and
says:
hey
I
want
this
great
new
feature
in
in
your
service.
A
I,
say
great,
come
to
me
with
a
design
and
then
I'll
tell
you
where
you
can
write
that
code
and
you
can.
You
can
start
contributing
that
code
in
and
a
lot
of
teams
have
done
that
we've
been
getting
very
meaningful
contributions
from
around
the
company.
That's
been
really
cool,
so
currently
our
repo
is
at
about
16,000
branches.
It's
still
about
two
gigs
in
the
working
directory
about
five
weeks
in
the
history.
A
You
know
I
consider
that
medium-sized,
like
I,
said,
but
we're
eyeing,
much
bigger,
repos
and
working
on
a
lot
of
things
in
that
space.
Overall,
we're
we're
on
a
journey.
You
know
we're
moving
towards
this
one
engineering
system.
We've
made
a
lot
of
progress.
There's
still
plenty
to
do
before
the
whole
company
is
is
on
board,
but
you
know
get
is
the
foundation
of
it
and
we're
working
hard
to
make
sure
that
it
scales
to
all
of
our
teams
and
make
sure
everyone
is
successful
in
in
using
using
git?
A
Well,
work
overdoing
on
large
repos
and
are
we
open
sourcing
we're
so
there's?
Let
me
just
talk
about
the
scale
problem.
I'm
gonna
be
a
little
bit
vague
on
it,
but
the
the
three
scale
problems
that
we
have
to
solve
is
the
number
of
branches
that
exist
in
a
repo,
because
that
affects
a
lot
of
operations
like
fetching
or
just
user,
like
local
operations,
that
a
user
does.
A
The
number
of
the
second
factor
is
the
number
of
people
trying
to
merge
into
a
branch
which
I
talked
about
before
and
then
the
third
is
the
the
size
of
the
of
the
repo.
You
know
we're.
We
are
planning
on
all
the
countries,
the
changes
that
the
improvements
that
we
make
in
those
areas.
We
are
certainly
planning
to
open
source.
Whatever
those
changes
are
that
affect
get
itself.
Some
of
the
changes
are
just
in
our
service.
Now
that
our
service
is
not
open
source,
so
those
don't
get
released
publicly,
but
any
changes
that
we
make.
A
Right,
so
how
are
we
measuring
improved
code
velocity
rather
than
just
talking
about
it,
anecdotally
and
yeah
velocity
and
quality?
Well,
actually,
we
were
working
on.
This
is
another
one
of
those
areas
where
different
teams
have
done
things
differently.
I
mean
working
on
unifying
it,
but
we
actually
have
a
fair
bit
of
telemetry
that
we
gather
from
our
get
users
and
we
we
can
measure
things
like
how
often
they're
successful
in
interacting
with
source
control.
A
You
know
we're
working
on
being
able
to
measure
like
how
many
times
they
actually
like
interact
locally
with
get
before
they
even
try
to
like
interact
with
the
service,
and
in
addition,
we
we
have
a
lot
of
telemetry
on
the
server
about
how
often
how
long
pull
requests
they
open.
How
often
they
fail
the
build
before
they're
emerged
in
like
how
many
iterations
it
takes
the
pass
and
gather
we
gather
data
on
how
often
the
build
breaks
once
it's
in
master,
because
that's
the
one
you
really
care
about
so
yeah.
A
Yep,
why
do
we
have
so
many
branches?
It's
because
we
tell
people,
branches
are,
or
your
friend
create
as
many
as
you
like,
and
we
have
300
or
so
people
creating
lots
of
topic
branches
for
all
the
work
items
they
have
they're,
not
necessarily
the
best
at
cleaning
them
up
afterwards
right.
So
one
recent
change,
recent
dish,
change
that
we
made
a
few
months
ago,
was
that
when
you
complete
a
pull
request,
we
now
automatically
delete
your
topic
branch
that
wasn't
always
there.