►
Description
Presented by Kate Ebneter, (ServiceNow).
Gerrit is a Git hosting service and code review tool that functions like GitHub with several key feature differences. ServiceNow recently migrated their entire development team and several other teams from Gerrit to GitHub Enterprise for internal code hosting and code review. Learn how the team solved—and continues to solve—their most pressing social and technical challenges.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Hi
I'm
Kate
up
neder
I
am
the
bill.
The
manager,
a
build
and
release
engineering
for
ServiceNow,
and
we
recently
migrated
from
Garrett,
which
is
a
tool
somewhat
similar
to
github
to
github,
Enterprise
and
I
was
asked
to
talk
a
little
bit
about
how
we
did
it
before
I
start.
I
would
actually
like
to
dedicate
this
presentation
to
two
people
who
were
very
important
to
it.
A
A
The
other
person
is
Abe
chin,
who
was
a
member
of
my
team
and
was
the
single
member
of
my
team,
most
responsible
for
dealing
with
all
the
testing
and
setting
up
and
everything
in
preparation
for
our
migration
from
Garrett
to
github,
and
he
passed
away
earlier
this
year
after
a
brief
illness.
Shortly
after
we
did
the
migration,
he
was
a
very
valuable
member
of
my
team
and
we
miss
him
a
lot
so
service.
Now,
as
you
probably
know,
does
software-as-a-service.
We
have
a
fairly
large
development
development
effort.
Development
environment
is
primarily
hosted
in
AWS.
A
The
tool
stack
that
we
use-
and
this
is
kind
of
important
for
this-
is
get
with
and
in
up
until
recently,
with
Garrett,
our
code
is
primarily
Java
and
JavaScript.
We
use
maven,
we
use
Jenkins
enterprise,
Nexus
Pro,
our
developers
typically
use
an
IDE
like
eclipse,
some
of
them
use,
IntelliJ
and
so
forth.
The
engineering
is
pretty
standard.
We
have
a
bunch
of
scrum
teams,
we
have
kind
of
two
divisions
in
our
development,
which
plays
a
part
in
how
we
have
to
deal
with
our
source
code.
A
We
have
a
platform
which
is
the
primary
product,
it's
basically
a
monolithic
application,
and
then
we
have
a
bunch
of
smaller
applications
that
can
run
on
top
of
that
platform
and
which
are
developed
primarily
in
JavaScript.
We've
experienced
extremely
rapid
growth.
Since
I
joined
the
company
in
2011
when
I
joined
there
were
approximately
50
developers
in
the
company.
We
have.
Let's
just
say
many,
many
many
times
that
number
today
the
company
went
from
being
a
100
million
dollar
revenue
company
when
I
joined
to
over
2
billion
in
revenue
today,
so
we've
grown
very
rapidly.
A
We've
we've
evolved
our
development
practices
very
rapidly
for
platform
development.
We
primarily
develop
on
feature
branches.
Now
we
originally
developed
on
master
that
got
ungainly
as
the
amount
of
stuff
we
were
adding
increased
over
time,
and
so
we
basically
went
to
these
feature
branches,
which
we
call
tracks,
and
there
are
approximately
70
of
them
that
we
have
to
mask.
A
We
have
to
marshal
every
night
and
build
and
test
applications
usually
develop
on
master
the
reason
I
mentioned
this
is
because
that
means
we
basically
have
to
significantly
different
things
going
on
in
our
source
code
repositories,
and
we
have
to
have
a
system
that
accommodates
both
of
them,
and
we
have
lots
and
lots
of
repositories,
lots
and
lots
of
repositories.
The
applications
typically
have
multiple
repositories,
one
for
the
application,
one
for
tests,
maybe
some
other
things.
A
I
guess
in
in
the
sense
that
that
particular
repo
has
caused
us
all
manner
of
issues
over
time
with
Garrett,
and
it
was
one
of
the
key
things
that
made
us
realize
that
we
needed
to
move
on
from
Garrett,
so
I'm
gonna
assume
that
most
of
you
are
not
familiar
with
Garrett.
So
I'll
tell
you
a
little
bit
about
it.
A
Garrett
we
chose
Garrett
because
at
the
time
in
2011
it
was
one
of
the
very
few
code
review
tools
that
actually
understood
get.
There
was
a
lot
of
code
review
tools
around
that
we're
just
trying
to
wrap
their
brains
around
get.
There
were
a
few
that
had
done
it,
but
not
very
well,
and
Garrett
was
actually
built
around
get
from
the
start.
It
was
developed
at
Google.
A
It
was
based
on
their
internal
code
review
tool.
It's
used
by
a
well
known
major
code
base,
namely
Android.
It
is
the
primary
repository
manager
used
by
the
Android
project
and
it's
open
source,
so
the
minuses
are
there's
no
pro
version
of
it
since
it's
a
Google
thing,
so
it's
not
like
Jenkins,
where
you
can
buy
Jenkins
Enterprise
and
it's
not
Nexus.
You
can't
find
Nexus,
Pro
or
github
Enterprise
or
whatever,
there's
no
pro
version.
A
So
there's
no
tech
support
the
only
tech
support
you
can
get
for
it
is
by
going
to
the
mailing
list,
begging
people,
those
kinds
of
things.
It
has
a
very
cumbersome
permissions
mechanism,
which
is
very
fine-grained,
and
you
can
do
a
lot
with
it.
But
it's
really
hard
to
understand
and
the
biggest
problem
we
had
was
its
performance
at
scale
and
its
scalability
when
we
started.
This
is
a
nice
little
picture
here
in
2012
we
started
with
a
small
garrett
server.
We
didn't
have
very
many
didn't
have
very
many
users.
Everything
was
fine.
A
By
2015,
the
number
of
developers
had
grown
quite
a
bit.
We
had
to
put
Garrett
on
a
bigger
server
by
2017
last
year
we
actually
were
running
to
Garrett
servers.
One
just
to
host
that
SNC
core
repo
that
I
mentioned
a
minute
ago,
and
we
had
lots
and
lots
and
lots
of
developers
and
the
servers
we
were
running
on
were
really
big
servers.
This
is
all
hosted
in
AWS
we're
using
ec2
instances.
A
This
is
the
standard
solution
to
a
Garrett
problem.
Give
it
more
memory.
The
server
that
we
were
running
Garrett
on
is
what
they
call
an
r
CR,
an
R
3.8
X
large
server.
It
has
32
virtual
CPUs
and
244
gigabytes
of
memory
and
believe
me,
it's
using
a
very
large
amount
of
that
memory
and
you're
probably
wondering
why
is
it
doing
that?
A
The
reason
is
that
it's
a
Java
monolith,
it's
a
multi-threaded
Java
process
and
it
handles
everything
handles
all
your
SSH
connections.
All
your
HTTP
connections,
all
your
interactive
sessions,
all
your
hook,
script
invitations
run
in
inside
of
the
the
Java
process.
It
handles
all
your
push
and
fetch
operations.
It
tries
to
handle
all
your
garbage
collection.
If
you
want
to
have
a
replica,
it
handles
all
the
replication.
It
does
all
this
inside
this
one
single
giant
blob.
A
A
A
We
discovered
that
because
Garrett
was
using
J
to
SSH,
the
SSH
connections
were
kind
of
painful
and
as
a
result,
we
switched
everybody
from
SSH
to
HTTP,
even
though
we
would
really
prefer
them
to
use
SSH
for
security
reasons.
We
discovered
that
Gerrit
couldn't
do
the
garbage
collection
on
some
of
our
larger
repositories,
in
particular
the
SNC
core
repo
it
would
just
fall
over.
A
We
put
the
SNC
core
repo
on
its
own
server,
which
was
exactly
the
same
as
the
server
that's
running
everything
else.
So
now
we
have
one
repo
sitting
on
a
server
with
244
gigabytes
of
memory
and
it
still
takes
20
minutes
to
clone,
and
then
we
finally
had
to
move
to
twice
weekly
maintenance
shutdowns,
because
the
the
garbage
collection
didn't
last
long
enough
and
we
had
to
clean
things
up
more
frequently
than
that.
So
that's
really
kind
of
a
an
overview
of
the
problems
we
were
having
now.
You
know
we
had
that.
A
We
had
that
slide
a
moment
ago.
That
said,
give
Garrett
more
memory,
so
Amazon
just
came
out
with
these
really
really
high
memory
instances,
so
they
have
like
up
to
12
terabytes
of
memory,
I,
okay,
that's
kind
of
probably
overdoing
it
a
little
bit,
so
we
decided
that
we
should
really
start
looking
at
something
to
replace
it.
A
We
needed
something
that
could
handle
multiple
simultaneous
clones
of
our
main
repository
without
falling
over
and
with,
preferably
without
requiring
huge
amounts
of
memory
in
the
process
we
needed
something
that
could
also
handle
the
standard
load
that
we
were
using.
Like
I
say
we
have
many
many
developers
all
over
the
world
who
are
hitting
this
server
all
the
time,
and
we
really
really
wanted
to
get
rid
of
having
to
have
these
maintenance
windows.
I
mean
having
maintenance
windows
twice
a
week
is
really
a
pain.
A
As
for
scalability,
we
really
wanted
something
that
could
scale
horizontally
as
well
as
vertically.
We
didn't
want
to
just
keep
throwing
memory
and
bigger
servers
at
the
problem.
We
wanted
to
be
able
to
have
replicas,
plural,
not
just
one
replica,
and
ideally
we
would
have
something
that
had
a
proven
track
record
of
being
able
to
scale
to
something
bigger
than
what
we
are
now,
preferably
many
times
bigger
than
what
we
are
now
as
for
features
they
weren't
as
important
as
these
other
things.
A
A
In
particular,
one
of
the
things
that
we
had
with
Garret-
and
this
was
one
thing
Garrett
was
really
good
at-
was
logging
everything,
so
we
literally
had
logs
of
everything
that
Garrett
had
done
so
every
push
every
poll,
every
HTTP
connection,
every
every
code
review
all
those
things
we
had
it
in
logs
and
we
were
able
to
take
that
and
generate
a
script
that
allowed
us
to
replay
a
busy
day
and
use
that
to
test
our
candidates
to
make
sure
that
they
could
handle
the
load.
And,
of
course
we
have
our
core
SNC
repository.
A
That's
really
critical
and
one
of
our
one
of
our
tests.
One
of
our
torture
tests
was
how
many
times
can
we
try
to
clone
this
simultaneously
before
the
candidate
falls
over
and
then
finally,
there's
the
user
experience
to
the
user
actually
like
it.
It
doesn't
do
us
any
good
if
we
get
something-
that's
performant
at
cetera,
et
cetera,
but
everybody
hates
to
use
it.
So
we
ended
up
with
two
primary
candidates.
A
A
We
did
have
some
teams
that
we're
using
github
github.com,
not
enterprise,
but
gitlab
had
a
lot
of
things
that
we
really
liked
about
it.
It's
an
open
source
code
base.
It's
they
use
the
same
code
as
their
sort
as
their
service
offering
it
has
an
integrated,
CI,
etc.
Their
tech
support
seemed
pretty
good.
A
They
give
you,
they
give
you
cool
socks
and
we
did
a
really
thorough
test
of
it
and,
and
it
actually
held
up
pretty
well,
we
were
pretty
impressed
with
it
and
I
hadn't
actually
considered
even
looking
a
github
Enterprise
initially,
because
when
I
first
looked
at
github
Enterprise
many
years
ago,
it
was
a
disaster.
The
github
folks
will
tell
you
this
too.
It
was
not
the
same
codebase.
It
was
something
they
threw
together
quickly.
They
eventually
realized.
That
was
a
big
mistake.
A
Well
about
a
year
ago,
I
went
to
Jenkins
world
in
San
Francisco
here
and
bumped
into
the
github
folks
and
I
made
a
comment
about
I
was
interested
in
github
Enterprise.
If
it
ran
the
same
code
as
they
run
on
their
site,
and
they
said,
oh,
it
does
so.
I
decided,
okay,
well,
we're
gonna
have
to
evaluate
that
too.
So
github
is
obviously
a
proprietary
code
base.
We
don't
get
to
peek
inside.
It
is
the
same
code
as
their
service.
A
A
A
So,
as
you
can
probably
guess,
we
ended
up
going
with
github,
Enterprise
or
I
wouldn't
be
giving
this
talk
here.
I'd
be
giving
it
at
whatever
get
labs
equivalent
is.
We
chose
github
Enterprise
for
a
number
of
reasons,
but
the
primary
one
was
that
git
lab
wants
to
be
everything
they
want
to
do.
Ci
they
wanted
to
do
deployment.
They
want
to
do
your
issue
tracking
blah
blah
blah,
which
is
great.
A
We
had
two
teams,
one
that
used
it
very
heavily
as
a
proof
of
concept,
and
we
had
another
team
that
was
some
of
our
senior
people
who
didn't
like
the
gitlab
interface,
which
is
kind
of
curious
because
they're
very
similar,
but
it
was
just
enough
different
that
they
preferred
the
github
experience
overall
and
the
other
thing
that
really
sealed
the
deal
was
I
was
talking
to
a
senior
manager.
Who
said
you
know
this
all
sounds
great,
but
I'm
still
really
concerned
about.
A
You
know
we're
just
gonna
have
the
same
problem
that
we
have
we've
Garrett,
that
you
know
next
year.
We're
gonna
have
to
buy
a
bigger
piece
of
hardware
to
run
it
on
we're.
Gonna
have
to
spend
a
lot
of
time
figuring
out
how
to
make
it
scale,
and
everything
and
I
said
well.
This
is
the
same
code
that
they
run
on
github.com
and
he
said:
oh
okay,
no
problem
buy
it.
So
those
were
the
primary
reasons
that
we
did,
that
the
issue
tracking
is
an
interesting
thing.
A
A
Issue
tracking
per
se,
but
it
turns
out
that
many
of
our
teams
actually
use
the
issue
tracker
in
github
as
a
communications
mechanism
to
discuss
various
things
well,
they're
working
on
problems,
so
they
use
it
for
design
discussions.
They
use
it
for
bug,
fix
discussions,
they
use
it
for
all
kinds
of
things
and
then
they
feed
all
of
that
back
into
their
work.
And
then
that
ends
up
going
into
the
PRB
reports
that
we
have
hosted
on
ServiceNow.
So
it
actually
does
help
our
workflow
quite
a
bit
to
have
that
capability.
A
A
My
team
practiced
the
migration
over
and
over
and
over
again
refining
the
run
list,
identifying
problems
with
it
and
going
through
and
finding
all
the
moving
parts
which
we'll
get
to
in
a
second
finding
solutions
for
every
issue
that
we
came
up
against
and
just
going
through
all
of
this
in
detail.
So
we
had
a
very
detailed
plan
not
just
for
the
day
but
for
all
those
steps
Lee
up
to
it
as
well.
We
had
training,
we
did
a
tech
talk
about
github
and
how
github
differed
from
Garrett.
A
In
terms
of
doing
code
reviews,
we
made
sure
that
we
had
people
who
were
able
to
support
github
in
our
different
locations
around
the
world
and
again,
like
I,
said
we
put
together.
This
very
detailed
run
list
of
everything
that
had
to
happen
on
the
day.
There's
a
lot
of
moving
parts
in
a
migration
like
this
there's
repos.
We
have
lots
and
lots
of
repos.
They
belong
to
lots
of
different
teams.
They
belong
to
different
parts
of
our
organization.
A
A
A
So
we
were
actually
able
to
separate
the
documentation
organization
from
the
development
organization
from
the
security
organization
from
you
know,
all
these
other
organizations,
and
then
we
also
can
have
our
business
units
and
our
scrum
teams
set
up
as
actual
teams
on
github.
So
that
was
very
useful.
We
had
to
train
ourselves
to
be
administrators
of
this
of
this
new
tool.
We
had
to
train
the
users.
A
We
wanted
to
create
a
separate
read-only
server
for
our
support
users.
Our
tech
support
tech
support
personnel
have
access
to
the
source
code,
so
they
can
understand
what's
going
on
in
ServiceNow
and
they
had
had
read-only
access
to
garrett.
We
created
a
separate
server
for
them
because
we
didn't
want
to
pay
for
an
additional
pile
of
licenses
for
them
to
be
perfectly
honest
with
you,
and
so
we
stood
up
a
separate
read-only
server
using
a
somewhat
less
sophisticated
tool.
A
A
We
actually
had
to
stand
back
and
say:
okay,
how
can
we
fix
this
and
we
actually
came
up
with
a
solution
to
fixing
that
which
ironically
kind
of
borrows
a
trick
from
Garrett,
because
what
Garrett
does,
when
you
create
a
code
reviewing
garrett
is
it
creates
a
special
ref,
the
it's
not
something
that
get
actually
knows
about
or
cares
about.
So
we
actually
did
the
same
thing
with
our
nightly
builds
that
we
tagged,
which
are
what
creating
all
of
these
references.
A
We
created
a
special
reference
that
get
doesn't
know
about
or
care
about,
and
so
it
doesn't
get
cloned
github
doesn't
try
to
show
it
in
the
UI,
but
we
can
clone
it.
You
can
copy
it
over
to
your
copy
with
some
simple
git
commands
if
you
need
to
and
finally
there's
all
these
integrations
with
all
our
other
tools,
we
had
to
make
sure
that
all
those
integrations
worked.
A
A
So
we
finally
got
everything
together:
we're
getting
ready
to
migrate,
we're
gonna
go
live.
This
is
scary,
it's
very
scary.
Shortly
before
the
migration
I
sent
out
an
email
to
everybody
telling
them
what's
happening.
This
was
not
like
spectacular
news
to
anybody,
because
everybody
was
waiting
for
this
with
bated
breath.
We
have
a
lot
of
people
who
were
really
unhappy
with
with
Garrett
and
its
performance.
A
We
did
get
one
slight
glitch,
which
was
management
said
this
isn't
going
to
work
unless
you
have
people
in
all
our
different
geographies
who
are
prepared
to
be
experts
to
help
people
if
they
have
questions,
because
one
of
our
big
development
centers
is
in
India
which,
as
you
probably
know,
is
literally
12
hours
difference
in
time.
You
know
so.
They're
like
you've
got
to
at
least
have
people
in
India
who
can
answer
questions
for
people,
while
you
guys
are
asleep.
So
we
delayed
the
employment
for
a
week.
A
A
A
A
A
Our
entire
QE
team
couldn't
log
in
and
we
were
like
what
why
the
QE
team
what's
going
on
here,
so
we
go
look
at
our
LDAP
integration.
Did
we
did
we
not
map
them
over
properly?
No,
that's
fine.
Everything's
fine
turned
out
that
our
IT
group
had
decided
that
because
for
some
reason,
Human
Resources
gave
the
QE
team
a
new
team
number.
A
A
A
You
can
log
on
to
it
your
usual
credentials.
Everything
is
fine.
It's
a
it's!
A
smaller
server!
It's
using
a
piece
of
open-source
software
called
GOG
x'
by
the
way
which
is
sort
of
a
lightweight
clone.
If
you
will
of
github
in
terms
of
its
its
appearance
and
everything,
it
has
a
very
similar
user
experience,
but
it
basically
works
out
very
well
for
that.
A
Finally,
some
conclusions,
so
the
takeaway
is
here
that
I
have.
Are
you
really
need
to
be
clear
on
the
reason
for
the
change?
I
think
that's
the
most
crucial
thing.
If
you
know
why
you're
making
the
change,
what
what
you
expect
as
the
outcome,
then
you
can
plan
around
everything
and
for
us
the
most
important
thing
was.
Can
we
clone
this
repo
that
we
have
in
a
reasonable
amount
of
time?
Do
we
expect
the
thing
to
be
able
to
scale?
Can
I
handle
the
load?
Those
are
the
things
that
are
really
critical
for
us.
A
We
actually
ran
a
test,
another
test
where
we
said
how
many
times
can
we
clone
this?
This
repo,
this
SNC
core
repo
simultaneously
before
github,
falls
over
and
the
answer
was
we
couldn't
find
a
number
that
it
would
fall
over
at
and
the
reason
is
it
would
get
to
a
certain
point
where
it
was
saying:
hey,
I'm
really
really
busy,
and
it
would
just
start
queuing
up
the
requests
instead
of
just
trying
to
serve
everything
all
at
once
and
dying.
It
would
do
something
intelligent
what
a
concept.
So
we
were
very
happy
with
that.
A
The
other
thing
is
our
India
team
is
ecstatic
about
the
performance,
because
the
typical
clone
time
for
this
repository
for
us
in
the
United
States
went
from
about
20
minutes
to
about
5
minutes
using
github
for
the
guidance
in
India.
It
literally
went
from
over
an
hour
to
approximately
20
minutes,
which
is
something
they
can
live
with
until
we
actually
get
them
a
replica
over
there.
A
geographic
repla
that
they
can
use,
which
is
one
of
our
next
steps,
that
we're
going
to
be
going
to.
A
The
other
takeaways
I
would
say,
is
have
really
good
realistic
tests
that
are
based
on
the
what
you're
doing
right
now,
because
we
had
all
these
logs
from
Garrett,
we
were
able
to
create
extremely
realistic
tests.
We
were
able
to
literally
replay
a
day's
load,
so
that
made
it
very
clear
whether
or
not
we
could
actually
use
this
tool
and
having
a
really
detailed
run
list
and
practicing
that
run
list
over
and
over
and
over
again
is
very,
very
helpful.
I
mean
we
literally
had
no
glitches
on
the
day
of
the
conversion.
A
They
just
went
through
the
run
list.
Everything
worked
because
they'd
practiced
it
so
many
times
it
was
really
clean
and
really
clear.
It
was
hands-down.
The
smoothest
transition,
I've
ever
been
involved
in
and
I've
been
doing,
building
in
and
release
engineering
for
over
25
years
here
in
Silicon
Valley,
so
I've
never
had
a
release.
Go
that
smoothly.
A
The
other
thing
is
communicate,
communicate,
communicate,
tell
people
what
you're
doing
tell
people
when
you're
gonna
do
it.
If
there
are
glitches,
tell
them
if
there
are
problems,
tell
them
but
communicate
to
your
user
base
and
make
sure
you
remember
your
entire
user
base,
instead
of
forgetting
your
support,
people
like
I
did
so.
A
A
Yeah,
so
the
question
is:
did
we
migrate
over
the
reviews
from
Gerrit?
Basically,
and
the
answer
is
no,
we
did
not.
We
can't
the
the
way
the
way
the
data
is
stored.
We
would
I
mean.
Let
me
rephrase
that
it's
not
that
we
can't.
It
said
it
would
be
a
huge
amount
of
work,
because
you
would
have
to
actually
literally
go
take
all
of
the
stuff
out
of
the
database
that
it's
stored
in
on
Gerrit
massaged
it
into
a
format
that
github
understood
and
put
it
over
there.
A
A
A
They
can
make
changes
since
they
own
certain
groups
of
repositories.
They
can
set
up
certain
rules
for
those
repos
and
design
their
workflow
around
their
set
of
repos,
in
a
way
that
they
could
not
do
one
garret,
and
so
we're
gradually
introducing
some
of
these
changes
over
time.
We
didn't
want
to
be
too
disruptive.