►
From YouTube: Ship happens: shipping pull requests at scale with the merge queue - GitHub Universe 2019
Description
Presented by Jack Li, Production Engineer at Shopify
At Shopify, developers merge and deploy over 400 pull requests each day to Shopify master. Two years ago, we built the merge queue as part of Shipit, an open source project, to control the thundering herd of merges and ensure ships happen quickly and safely. In this talk, Shopify will explore the design of the first iteration of the merge queue, what they've learned from that experience, and how they're evolving the merge queue to continuously scale developer productivity.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Good
afternoon,
everybody
welcome
to
ship
happens,
shipping
PRS
at
scale
with
the
merge
queue.
My
name
is
Jack
Lee
and
I
am
a
production
engineer
at
Shopify
on
the
developer,
acceleration
team.
So
broadly,
we
commit
our
time
to
helping
our
developers
to
be
productive
and
build
the
tools
and
infrastructure
to
help
keep
them
happy.
A
So
in
case
you've
never
heard
of
Shopify
Shopify
is
the
leading
multi-channel
commerce
platform
that
powers
more
than
1
million
merchants
globally.
Some
of
our
merchants
include
the
get
up
store.
The
get
up
shop
out
front
is
actually
using
our
POS
system,
also
fashion
OVA
and
Kylie
cosmetics,
just
to
name
a
few.
As
you
can
imagine,
next
week's
Black
Friday
it's
gonna
be
a
fun
time
for
us,
so
the
core
of
Shopify
is
built
on
a
large,
majestic
Ruby
on
Rails
monolith.
A
So
today,
I'll
be
talking
about
the
merge
queue,
which
is
a
tool
that
we
created
to
maintain
high
velocity
of
shipping
at
scale.
I
also
want
to
share
our
release
philosophy
and
how
that
shaped,
the
tooling
that
we
build
so
I,
don't
have
a
fun
github
action
for
you
guys
to
take
home
today
and
immediately
take
advantage
of
all
of
our
work
and
I.
A
A
A
The
first
rule
is
a
master,
must
always
be
green,
so
we
want
to
be
able
to
deploy
the
master
at
all
times
and
if
master
is
in
green,
then
we
have
to
get
it
back
to
green
as
fast
as
we
can
master
should
be
close
to
what
we
run
in
production.
So
we
believe
that,
no
matter
how
much
testing
you
do,
the
ultimate
test
is
to
see
how
the
changes
behave
with
production,
traffic
and
production
data.
So
when
we
leave
things
on
deploy
the
master,
this
increases
the
risk.
The
next
deploy.
A
A
Our
developer
workflow
looked
a
little
bit.
Something
like
this.
So
developer,
go
ahead
and
create
a
new
poll
request,
increases
dog
bites
by
200%.
Would
that
be
nice
and
then
we
will
wait
for
branch
CI,
and
this
is
SIA
that
just
runs
on
whatever
the
state
of
the
branches
in
2016.
This
took
about
15
minutes
to
run
and
eventually
it
passes
sweet.
We
can
now
merge
a
master
by
pressing
our
favorite
button.
The
merge
pull
request
button
once
this
is
done,
we
can
deploy
to
production
right
nice
and
easy,
not
exactly
so.
A
At
this
point
we
were
starting
to
get
a
little
big
and
we
had
lots
of
developers,
shipping
lots
of
changes.
So
while
we
were
waiting
for
our
CI,
many
others
were
waiting
for
theirs
as
well
and
after
we
merged
they
came
into
master
right
after
us
now,
while
each
of
these
poor
requests
have
passed
CI
individually
on
their
branches,
they
aren't
guaranteed
to
pass
once
we
actually
put
them
together,
but
not
a
problem
to
ensure
that
we
can
integrate
these
changes
together
properly.
A
A
At
this
point,
we
have
a
continuous
delivery
tool
called
ship
it,
which
is
an
open
source
tool
that
we
created
here
at
Shopify
and
the
way
that
ship
it
works
is
that
it
automatically
goes
ahead
and
deploys
batches
of
past
pour
quests
out
to
production.
So,
in
this
case
here
it
would
see
the
green
statuses
and
it
would
automatically
craft
a
batch
of
all
of
the
undeployed
commits
up
to
the
last
green
and
it
will
go
ahead
and
deploy
them
to
production.
A
We
do
perform
canary,
deploys
and
multiple
deploy
steps
by,
for
sake
of
simplicity,
I'm,
just
gonna
group,
the
whole
thing
as
production
and
then
after
the
first
deploy,
we
send
out.
We
get
the
next
to
poor
requests
to
pass
as
well.
No
problem
ship
is
gonna,
go
ahead
package,
those
up
and
deploy
those
out
to
production
as
well,
and
this
helps
make
it
that
we
don't
need
to
coordinate,
deploys
and
instead,
all
of
this
stuff
happens
automatically.
A
A
The
first
problem
we
have
is
with
conflicts
and
I'm,
not
talking
about
hard
conflicts
where
you're
you
and
someone
else,
our
team,
the
same
line
of
code
but
I'm
talking
about
soft
conflicts
which
are
much
harder
to
detect.
These
are
changes
that
might
pass
CI
independently,
but
they'll
fail
once
they're
put
together.
A
A
Uh-Oh,
but
how
can
this
happen
if
it
already
passed
the
branch
CI?
So
the
first
answer
is:
maybe
it
wasn't
rebased
properly?
Maybe
the
branch
was
based
off
of
was
master
from
last
week
that
could've
caused
it,
or
maybe
this
can
happen
because
of
saw
conflicts.
So
maybe
four
and
four
or
four
one
five
over
there
introduced
some
change
that
conflicted
logically
with
four
one:
six
causing
it
not
to
pass
one
and
still
together
and
because
four
one
six
is
broken,
the
build
everything
else
after
it
would
fail
as
well.
A
Now
this
is
not
the
end
of
the
world.
We
can
still
make
some
progress,
so
this
point
ship
is
gonna,
go
long
and
see.
Okay,
so
I
have
some
passes.
Sorry,
and
so
we
can
still
deploy
the
first
batch.
But
after
we
do
that,
unfortunately
we
get
stuck
and
to
make
matters
worse.
Other
people
might
not
realize
the
master
is
broken
right
now
and
they're
gonna
continue
merging.
A
A
So,
every
time
the
conflict
shows
up
master
ends
up
further
and
further
ahead
of
what
is
running
in
production
right
now,
and
we
call
this
phenomenon
drift
and
it
substantially
increases
the
risk
of
the
next
deployment.
If
this
last
batch
deploys
and
something
goes
wrong,
we
now
have
eleven
potential
pull
requests
that
could
have
caused
the
problem.
In
practice,
this
number
is
even
larger
due
to
how
fast
our
proquest
our
developers
ship
their
pull
requests
so
to
an
in
order
to
ensure
that
we
can
deploy
safely.
A
We
can't
deploy
fast
enough
and
now
we
have
to
go
ahead
and
deploy
a
large
batch
of
changes
to
go
out
just
so.
The
emergency
fix
can
make
its
way
out
now.
We've
just
introduced
a
lot
of
risk
and
after
an
emergency
fix,
it's
possible
that
one
of
the
other
ten
pull
requests
might
have
introduced
a
different
problem,
and
now
we
are
in
a
cycle
or
we're
constantly
fixing
new
failures.
A
A
But
on
the
next
Tuesday
we
have
to
merge
all
of
the
changes
that
developers
have
been
working
on
during
the
code
freeze
and
in
order
to
coordinate
this
without
flooding
master
again
to
prevent
drift.
We
asked
developers
to
add
their
poor
request
to
a
milestone,
and
then
we
will
go
ahead
and
we
will
slowly
merge
a
pro
request
and
make
sure
they
come
out
and
save
batches.
A
A
A
We
created
a
Chrome
extension
that
adds
this
add
to
merge
queue
button
to
every
single
pull
request
so
that
the
experience
fill
feels
familiar
to
the
github
experience
and
when
that
button
is
pressed,
the
PR
gets
added
to
a
queue
in
a
system
that
we
maintain
and
just
like.
In
the
previous
case,
everyone
else
can
merge
the
proto
class
as
well
and
they'll
end
up
in
the
same
queue.
A
The
merge
queue
here
is
going
to
merge
three
of
the
poor
quest
to
master
and
that
the
other
to
sit
in
the
queue
and
once
the
pork
ass
get
merged.
That
is
when
we
start
to
master
CI,
so
this
part
uses
the
same
logic
as
a
previous
solution
where
we
go
ahead
and
run
CI
once
it's
emerged
and
things
will
start
deploying
to
production
once
this
CI
succeeds.
A
Now
one
very
useful
feature
of
this
merge
queue
is
the
ability
to
lock
or
pause
merges
temporarily.
So
a
thing
back
to
the
emergency
case.
If
an
emergency
poor
request
needs
to
go
out
right
now,
we
can
make
sure
that
it
can
go
out
in
isolation
or
with
very
little
other.
Changes
in
this
helps
mitigate
risk,
and
this
very
simple
mechanism
worked
very
well
and
they
gave
us
a
lot
of
control
over
the
rate
in
which
we
were
merging
to
master.
A
Now,
while
we
wait
for
master
CI
on
the
3
protists
that
we
merged
the
other
paw
requests
are
sitting
in
the
queue
basically
in
limbo,
and
if
we
assume
that
our
CI
takes
30
minutes
to
run.
This
means
that
the
authors
of
4,
1,
7
and
4
1
8
over
there
have
to
wait.
30
minutes
before
CI
even
starts
on
their
pour
requests.
There's
also
no
guarantee
that
after
they
weigh
those
30
minutes
and
they
merge
to
master
that
see,
I
would
pass
after
that.
A
So
it
could
take
up
to
an
hour
in
this
case,
just
for
them
to
find
out
that
they
had
an
issue,
and
if
you
were
later
down
in
the
queue
there's
more
per
class
in
there,
you
would
have
to
wait
even
longer
hour
and
a
half
two
hours
three
hours.
You
never
know,
and
the
second
problem
was
that
we
had
this
browser
extension
now
that
we
had
to
maintain.
A
A
A
First
thing
we
did
was
we
moved
to
a
new
way
of
interaction
with
the
queue
using
comments,
so
in
every
pull
request.
Developers
are
greeted
by
a
message
from
our
bot
and
I
could
give
some
simple
instructions
on
how
to
use
the
Q
and
when
developers
are
ready
to
share
their
pull
requests.
They
can
issue
the
slash
shipit
comment
to
the
poor
request:
we'll
go
ahead,
receive
the
webhook
and
we'll
run
a
few
basic
checks.
A
We
want
to
make
sure
that
branch
CI
has
passed
and
that
reviewers
have
approved
the
pull
request
and
at
the
checks
pass
we
go
ahead
and
add
the
poor
class
to
Q
and
we
throw
in
little
thumbs
up
emoji
to
get
some
feedback.
If
the
checks
fail,
we
will
go
ahead
and
add
a
comment
to
the
pull
request
explaining
the
problem.
A
So
our
new
pipeline
now
looks
a
little
bit
like
this
/.
If
it
on
a
poor
requests,
now
adds
it
to
our
new
merge
queue
but
at
the
same
time
we're
not
doing
something
smart
with
the
internals
of
the
queue
as
well.
So,
let's
zoom
in
on
that
a
little
bit
so
here,
I
have
the
inside
of
the
second
merge
queue
inside
of
this
queue.
A
So
how
do
we
get
this
stuff
onto
master
so
this
time,
instead
of
going
ahead
and
merged
in
branches
and
stuff
into
master?
We're
gonna
instead
fast
forward?
Our
master
branch
to
point
to
the
last
green
merge
commit
so
notice
over
here
master
one
from
zero,
zero
zero
and
pointed
directly
to
the
Shah
of
the
last
passing
for
requests,
burst,
commit
which
is
bbbbbb,
and
this
allows
us
to
preserve
the
predictor
branch,
ensuring
that
we
don't
throw
away
any
of
the
in
progress.
A
A
A
So
in
this
case
the
queue
is
locked,
but
the
icons
back
green
for
our
commits
the
queue
is
still
locked.
So
we
can't
merge
anything,
but
that's
okay,
because
once
things
get
unlocked,
we
can
go
ahead
and
ship.
Those
changes
immediately,
so
problem
gets
fixed,
the
lock
is
green
and
we
can
go
ahead
and
immediately
fast-forward
those
commits
into
master
and
have
them
start
deploying
here's
a
graph
of
the
mercy
size
after
we
launched
the
new
merge
queue.
A
So
the
first
vertical
line
you
see
is
when
the
queue
got
locked
for
an
incident
you
can
kind
of
see
after
the
log
happened.
The
queue
starts
growing
because
of
the
pull
requests
that
our
developers
have
been,
adding
to
the
merge
queue
at
the
second
red
line.
The
queue
gets
unlocked
and
immediately.
We
can
go
ahead
drain
and
merge
all
of
those
poor
quests
into
master
and
have
them
start
deploying
right
away.
A
A
Unfortunately,
it's
not
so
simple
in
our
case
here
removals
become
very
expensive,
because
if
we
remove
a
pull
request
from
the
queue
and
means
that
the
structure
of
the
entire
branch
changes-
and
so
we
actually
have
to
throw
away
all
of
the
work
for
all
of
the
protests
behind
it
also
a
single
CI
failure
could
be
a
very
bad
signal.
It's
possible
to
have
infrastructure
issues
with
CI,
we've
all
seen
those
maybe
flaky
tests.
A
A
So
the
simple
solution
that
we
went
with,
which
introduced
a
concept
called
the
failure,
turns-
and
this
is
a
configurable
per
repo
value
that
basically
says
how
many
successive
failures
do
you
need
before
you
kick
something
out
from
the
queue
in
our
example.
Here
we
can
assume
a
failure,
tolerance
of
two
which
we
can
show
using
the
red
line.
This
means
that
we
can
tolerate
two
failures,
but
on
the
third
failure,
we're
gonna
go
ahead
and
eject
for
1/6
from
the
queue.
A
To
do
this,
we
created
a
special
command
called
slash,
shipit
emergency
and
this
functions
just
like
the
default.
Merge,
pull
request
button
by
issuing
this
command
we're
going
to
go
ahead
and
merge
that
pull
request
directly
into
master
and
once
that
Perkis
lands
on
master,
it's
gonna
go
ahead
and
start
its
own
CI.
At
this
point,
developers
can
whit
can
decide
if
they
want
to
wait
flat,
CI
to
finish
or
if
they
want
to
just
go
ahead
and
deploy
immediately.
A
So
as
an
example,
let's
assume
that
our
tree
looks
like
this
again
to
second
the
previous
example
and
this
time
uh-oh
we
got
a
problem,
so
something
has
broken
production
and
now
there's
exceptions
everywhere,
but
don't
worry.
This
is
fine.
A
fix
is
created
for
the
production
problem
and
we
can
go
ahead
and
slash
ship.
It
emergency
to
land
it
directly
into
master,
so
we're
here.
The
master
shop,
changes
and
CI
starts
for
the
commit
we
just
merged
in
and
now
because
master
has
changed.
A
Our
predictive
branch
has
now
diverged
and
is
no
longer
compatible
with
the
current
master.
So
what
this
means
is
we
have
to
go
ahead
and
create
a
new
one
and
reset
CI
for
everything
in
the
queue.
So,
even
though
it's
possible
to
ship
things
during
emergencies,
and
now
it
becomes
a
very
disruptive
process.
A
A
Evergreen
master,
so
since
we
are
only
forwarding
to
commits
with
passing
builds
master,
will
only
ever
go
from
passing
state
to
passing
state.
This
helps
local
development,
so
developers
will
never
pull
a
failing
master
and
will
never
have
to
coordinate
fixing
the
master
build
ever
again
ci
before
merge.
So
because
we
now
build
CI
in
the
queue
we
can
continue
making
progress
even
without
things
explicitly
landing
and
master,
and
this
gives
us
higher
throughput
in
times
when
the
queue
is
locked
or
one
master
is
getting
too
far
ahead.
A
Automatic
removal
of
failed
PRS
in
the
queue.
So
now
we
don't
need
humans
to
notice
our
problems
and
instead
we
can
go
ahead
and
message
offending
authors
directly
using
robots.
This
means
less
work
for
developers
as
a
whole
and
there's
also
no
more
stress
that
the
author
is
holding
everyone
up
here.
A
Some
trade-offs,
removing
a
pair
from
the
queue
now
resets,
the
AI
statuses.
So
we
talked
a
little
about
that.
Removing
is
now
very
disruptive
for
now.
We're
okay,
with
this
trade-off
since
breakages
are
still
fairly
uncommon.
But
if
too
many
rules
happen,
this
the
queue
will
start
on
dramatically
and
we
have
to
revisit
this
problem
and
skipping
the
queue
resets,
our
CI
statuses,
the
same
problem
as
the
previous
trade-off.
The
main
case
here
is
emergency
merges
and
they
are
very
disruptive
because
they
reset
CI
for
everyone.
A
So
what's
next,
so
obviously
we
don't
address
on
the
trade-offs
that
are
current
solution
has,
and
one
of
the
things
we're
thinking
about
is
this
idea
called
parallel
trees
so
right
now,
the
reason
why
we
have
lots
of
shortcomings
that
we
have
is
because
we
have
a
single
predictive
branch
at
any
given
time.
But
what
if
we
implement
a
parallel
tree
approach
where
we
build
multiple
realities
of
what
master
could
be?
A
So
in
this
example,
here
we
had
the
5
Pro
requests
we're
talking
about
earlier,
and
we
have
one
tree
that
includes
all
5
of
them
and
then
we
build
5
additional
trees
where
each
of
those
additional
trees
is
missing.
One
of
the
poor
requests
out
of
the
first
5,
so
we
call
this
the
one
level
of
exclusion
where
basically
we're
saying
that,
and
most
only
one
PR
will
be
broken
at
a
time.
In
the
most
ideal
world.
A
A
This
makes
it
so
that,
if
there
is
a
problem,
we
can
still
make
progress
now
we
did
try
running
this
model
in
production
for
about
a
few
days,
and
we
realized
that
even
with
this
only
one
level
of
exclusion,
we
were
incurring
too
much
load
so
going
to
the
next
year.
We
want
to
find
a
way
to
do
this.
That's
smart
and
cheaper
for
us
to
run.
A
Faster
CI,
so
we're
also
thinking
about
making
CI
faster
because
well
that's
kind
of
the
part.
That's
bounding
us
right
now.
If
we
can
make
CI
faster,
then
the
worst
cases
in
our
system
won't
be
as
bad
and
everything
becomes
faster
as
a
whole.
So
spoiler
we
do
have
a
team
working
internally
on
this
problem,
specifically
they're
looking
and
running
a
subset
of
our
test
suite
each
time.
I
said
the
whole
thing,
but
we're
still
pretty
early
into
this,
and
so
we
don't
have
anything
yet.
A
But
we
will
share
our
findings
when
we
can
so
earlier
today
we
did
release
a
more
technical
blog
post
to
our
engineering
blog
about
the
design
of
the
actual
key
we
built,
and
if
you
want
to
learn
more
about
what
we're
doing
what
we
have
done
check
out
our
engineering
blog
at
engineering
da
choppah,
5,
calm
and,
to
be
honest,
it's
been
hard
for
us
to
find
other
solving
problems
in
this
space
and
scale.
But
if
that
does
sound
like
you,
let's
talk
so
me
and
my
colleagues
darren
and
willem
over.