►
Description
Presented by
Keith Ballinger, SVP of Engineering
Keerthana ""KK"" Kumar, Staff Engineering Manager
Liz Saling, Director of Engineering
GitHub invested an entire quarter to identify, prioritize, and tackle some of the business’ biggest challenges both inside and out. Hear from Keith Ballinger, SVP of Engineering, Liz Saling, Director of Engineering, and Keerthana “KK” Kumar, Staff Engineering Manager, how they planned, approached, and solved an initiative that involved over 400 engineers, 3 categories of work, and 20 specific projects.
For more from GitHub Universe 2020, visit https://githubuniverse.com
As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub
Thanks!
Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github
About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com
A
A
A
A
A
One
was
around
both
the
outer
loop,
how
you
deploy
software,
but
also
our
observability,
both
observability
during
a
deploy
and
then
just
generally
speaking,
another
big
set
of
work
streams
was
around
the
inner
loop
and
testing.
That
is
when
someone
is
coding,
they're,
making
a
build
locally,
they're
running
tests,
all
of
those
kinds
of
things,
and
then
the
final
loop
was
final
work
stream
was
specifically
around
reliability.
A
What
parts
of
the
system
do
we
need
to
make
better
and
stronger
so
that
you
know
the
site
is
always
up
always
serving
our
customers
in
all
cases,
and
so
we
organized
across
those
we
had
objective
leads,
who
kind
of
looked
at
each
one
of
these
three
and
then
underneath
them.
We
created
virtual
teams
of
work
stream,
leads
to
tackle
very
specific
problems,
and
so
we're
going
to
kind
of
go
through
that
funnel
with
you
today
and
I'm
going
to
hand
it
off
to
liz
salling.
A
C
C
So
we've
been
tweaking
this
process
over
time
to
make
it
better
and
given
that
we
have
this
opportunity
to
do
so
much
more
together,
we
asked
all
of
engineering
for
their
ideas
and
proposals,
and
we
took
some
time
to
evaluate
each
one
of
those
for
the
value
and
impact
it
could
have
on
our
deployment
process.
And
you
know
we
had
to
make
some
hard
decisions.
We
had
a
lot
of
good
proposals
and
ultimately,
we
selected
these
as
some
of
the
work
streams
that
the
projects
that
we
were
going
to
work
on.
C
First,
we
wanted
to
look
at
how
we
could
speed
up
our
artifact
builds
and
we
need
some
more
progressive,
rollout
stages
and
automation
around
them,
and
we
noticed
that
there
were
recurring
themes
of
deployment
problems
that
we
could
eliminate.
That
could
take
care
of
that
need
for
manual
intervention.
C
So
we
aimed
to
increase
that
deployment
success
rate
and
we
also
wanted
to
add
some
resilience
in
our
deployment
process
to
github.com
outages
so
that
we
could
use
github.com
more
in
our
deployment
process
so
that
we
could
restore
service
faster
when
there
is
in
fact
an
event
all
right.
So
once
we
settled
on
what
we
were
going
to
do,
we
took
a
minute
to
settle
together
how
we
were
going
to
do
it.
Here's
some
of
the
things
that
we
did
that
I
thought
really
contributed
to
this
going
well
for
us.
C
We
extended
this
opportunity
to
a
lot
of
different
individuals
and
it
worked
given
what
we
outlined
and
how
we
kept
up
with
each
other
and
that
the
support
that
we
had
from
the
other
leaders
and
around
the
engineering
organization
right.
So
the
first
thing
leads
had
to
officially
do
was
take
a
couple
weeks
and
come
up
with
a
project
plan
so
to
set
them
up
for
success.
We
were
really
particular
that
they
had
to
work
with
one
or
two
other
people
that
fit
a
few
perspectives
so
that
we
could
fight
bias
right
from
the
start.
C
Of
course,
we
needed
someone
who
had
that
context
right.
Where
are
things
at
now?
What's
been
tried
before?
What
pitfalls
do
we
need
to
look
out
for
the
next
key
person
in
here
was
someone
who
maybe
wasn't
as
familiar
with
the
current
systems,
but
they
had
other
relative
expert
expertise,
experience
ideas
and
not
the
least
important
was
that
if
this
wasn't
already
covered
with
the
other
two,
we
had
to
have
a
representative
from
the
team
that
would
be
picking
up
this
work
and
continuing
to
maintain
it.
C
So
with
that
we
kicked
off
the
work
we
ran
the
project
like
we
typically
would
and
again
here's
a
few
key
points
in
running
the
project
that
really
helped.
It
really
helped
that
thing
for
us.
We
reminded
each
other
often
to
keep
that
healthy
dissatisfaction
with
the
status
quo.
So
instead
of
running
each
project
separately,
you
know
the
leads
met
together,
often
especially
right
up
front
right.
We
and
then
at
least
once
a
week
after
that-
and
we
talk
about
where
we
could
combine
efforts
and
where
can
we
build
off
each
other's
outcomes?
C
C
We
just
planned
to
get
to
it
later,
but
we
were
able
to
pull
that
in
and
get
that
going
quickly
and
we
did
more
than
what
we'd
originally
scoped
for
separating
the
merge
queue
out
of
the
deployment
process
that
ended
up
cutting
a
big
chunk
of
work
that
we
were
planning
to
do
for
deployment
success
rates.
So
just
to
reiterate
on
that
healthy
part
of
the
dissatisfaction
with
this
status
quo
right,
there's
a
natural
tendency
to
focus
on
friction
in
the
current
system.
C
It
was
great
to
see
people
really
acknowledging
the
work
done
and
the
people
who
did
it
and
the
dreams
that
they
had
for
the
things
that
if
they
had
all
the
time
and
people
available
that
they
could
have
done,
but
we
really
drove
home
that
help
don't
judge
thing
consistently
and
it
was
great
to
see
that
throughout
because
really
we
wanted
everyone
to
focus
on
their
well-being,
especially
this
year,
but
always
right
even
getting
started
with
us.
We
acknowledge
how
disruptive
it
was
compared
to
continuing
work.
C
We
usually
focus
on
and
taking
that
on
this
year
it
wasn't
a
decision
that
we
made
lightly
and
we
tried
to
do
some
things
to
make
it
a
little
more
fun
right,
like
we
worked
on
getting
some
swag
together
and
there
may
have
been
some
fun
internal
evidence
that
we
produced,
but
really
we
did
focus
on
working
together.
Well,
so,
as
we
progress
through
the
summer
as
usual,
you
know
we
we
learned
some
things
we
had
to
adjust.
We
had
to
accommodate
new
information
and
situations
that
came
up.
C
We
ended
up
dropping
some
work
streams
that
we
have
originally
planned
to
do
so
that
we
could
really
focus
in
and
do
the
ones
that
we
focus
on.
Well
again,
we
didn't
want
to
squeeze
ourselves.
We
really
wanted
to
focus
that
energy.
While
we
had
everybody's
attention
and
move
the
biggest
things
forward
together,
and
we
also
identified
that
we
were
generating
a
lot
of
work
for
other
teams
to
implement
that
fan
out.
Work
to
you
know
implement
the
improvements
across
the
organization.
C
C
Okay,
so
fast
forward
a
couple
of
months
and
here's
what
we
were
able
to
accomplish.
First,
we
added
several
reliability
features
and
thankfully
we
were
able
to
maintain.
We
were
able
to
do
this
without
adding
friction
right.
We
were
able
to
maintain
that
critical
cue
to
deploy
time
that
we
watch,
because
we
shave
time
off
elsewhere
and
we've
seen
our
deployment
frequency
bump
up
we're
often
delivering
over
100
pull
requests
a
day
to
github.com.
Now
the
number
of
deployment
rollbacks
we
have
to
do
are
fewer
and
fewer.
C
C
Last,
let's
see
the
time
to
resolve
incidents
when
they
do
happen
is
vastly
improved,
and
I
want
to
mention
a
thing
that
I
was
super
excited
to
see
happen
here.
We
set
up
a
mirror
of
our
github
repository
so
that
we
can
build
and
deploy
github
still
when
github
isn't
available
the
thing
that
we
do
it
when
it
is
and
another
one
I'm
super
excited
about,
is
that
we're
enabling
merge
queues
for
more
of
our
repositories.
C
We've
separated
out
how
we
bundle
up
our
pull
requests
and
that's
separate
out
of
the
deployment
process
now
for
that
repository
so
yeah.
So
all
of
this,
along
with
some
simpler
confidence,
dashboards
again
that
progressive
rollout
automation
and
really,
I
believe,
just
involving
so
many
more
people
to
work
on
this
experience
and
the
outer
loop
and
the
observability
it
led
to
this
great
improvement
in
our
development
happiness
measurement
that
we
take
each
quarter.
C
160
percent
y'all,
like
that's,
that's
big,
and
I
don't
want
to
like
to
call
out
another
key
observation
related
to
this,
and
that
is
the
increased
unity
across
the
engineering
organizations
right.
So
many
teams
had
opportunities
to
work
together
on
things
that
we
wouldn't
ever
get
a
chance
to
collaborate
on
normally
and
the
results
of
that
have
been
so
positive,
so
yeah.
So
these
results
right
there
they're
fun,
they're,
really
good,
but
really,
how
do
we
keep
this
up?
C
That's
what's
really
important,
so
we
have
now
what
I
like
to
call
anti-slip
measures
in
place
formally
here.
We
call
it
our
fundamentals
program,
but
for
every
improvement
that
we
make,
we
set
new
benchmarks.
We
raise
that
bar.
We
monitor
them
and
tweet
to
understand.
Does
something
need
to
be
nudged
a
bit
right.
C
We
have
that
additional
visibility
to
make
it
a
lot
easier
to
have
these
conversations
with
product
managers
and
engineering
leaders,
and
it
makes
planning
able
to
factor
this
in
better
and
at
least
be
more
aware
of
what
we're
going
to
set
aside.
If
we're
choosing
to
do
other
things
right
and
we
continuing
and
we
continue
partnering
and
working
with
other
teams
who
would
really
like
to
contribute-
and
this
is
helping
us
work-
smarter
and
be
more
effective
and
really
get
that
high
performing
thing
without
having
to
work
longer
hours.
C
B
Thank
you,
liz
everybody,
I'm
kk
and
I'm
a
staff
engineering
manager,
but
specifically
on
this
effort
to
pay
down
technical
debt.
I
was
one
of
the
work
stream
leads.
You
know
as
keith
and
liz
mentioned,
you
know
as
we
build
software
for
a
long
time
as
just
how
we
have
done
in
github.
We
accumulate
a
ton
of
technical
debt.
So,
right
before
we
started
this
initiative,
we
asked
the
entire
engineering
org
to
suggest
you
know
what
kind
of
engineering
improvements
we
want
to
see.
B
So,
let's
talk
about
what
goes
behind
the
scene
as
we
build
github.
So
github.com
is
supported
by
a
large
ruby
monolith
that
is
deployed
several
times
a
day.
As
an
engineering
team,
we
are
reviewing
and
shipping
hundreds
of
pulled
requests
a
day
and
at
the
same
time,
so
we
also
care
about
quality
and
maintainability,
and
that's
why
our
project
contains
about
seven
thousand
test
suites
and
five
5000
test
files.
B
So
before
every
deployment
and
after
every
commit
on
an
active
pull
request,
we
run
all
of
our
continuous
integration
jobs
to
ensure
quality.
These
jobs
are
foreign
type,
so
there's
build
jobs,
linting
jobs,
unit,
test
jobs
and
integration,
jobs.
In
total.
We
have
about
25
ci
jobs
together
they
take
about
45
minutes
to
run
well.
45
minutes
is
a
long
time
to
wait
and
I
often
could
get
a
workout
in
before
you
know.
B
Things
are
being
ready
for
merge,
but
I
also
want
to
note
that
the
in
the
best
case
scenario,
assuming
all
ci
jobs
were
successful
around
the
first
run
the
lead
time
to
change
that
liz
was
talking
about.
That
is
the
time
it
takes
for
a
piece
of
code
to
go
from
a
developer's
commit
to
a
feature
on
the
website,
took
about
two
hours
and
that's
the
best
case
scenario.
B
Of
course,
we
also
had
break
class
scenarios
in
place
in
order
to
ship
hot
fixes
to
get
around
this
45-minute
ci
runtime,
but
the
norm
was
45-minute
and
it
is
evident
that
it
slowed
down
our
development
process
and
added
to
developer
friction.
So
going
back
to
that
25
ci
jobs,
we
looked
deeper
into
it
and
found
out
that
all
but
two
integration
jobs
took
less
than
13
minutes.
So
obviously
the
two
integration
jobs
were
considered
a
bottleneck
in
our
deployment
process
and
a
little
more
into
that
two
integration
jobs.
B
We
were
actually
running
tests
on
our
github
enterprise
server
builds.
So
you
know
what
does
github
enterprise
server.
Maybe
you
used
it
as
part
of
enterprise
teams,
but
we
ship
github
enterprise
server
every
quarter
and
we
make
a
patch
release
every
two
weeks,
but
on
the
contrary,
we
ship
dot
com
several
times
a
day.
So
we
wanted
to
find
the
solution
that
can
guarantee
the
quality
of
every
product
we
ship,
while
still
keeping
the
ci
runtime
as
low
as
possible.
B
So
we
developed
the
solution
calling
it
deferred
compliance.
Different
compliance
is
basically
a
tool
that
was
integrated
along
with
our
ci
workflow.
This
allowed
our
developers
to
continue
to
ship
updatesthe.com,
which
is
github.com
without
having
to
wait
for
the
results
of
the
long-running
ci
jobs
that
tested
github
enterprise
server.
We
essentially
deferred
that
for
a
later
time,
let's
walk
through
a
scenario.
B
A
developer,
let's
say,
creates
a
pull.
Request,
gets
everything
reviewed
and
it's
set
to
merge.
At
this
point,
we
set
aside
two
different
types
of
jobs,
one
the
long-running
ci
jobs
that
run
in
the
background
and
two,
the
short
runnings,
quick
ci
jobs
that
are
essential.
So
as
soon
as
all
the
essential
nci
jobs
are
complete,
the
pulled
request
is
set
to
merge
and
deployed
on
to
dot
com.
Now,
at
the
same
time
as
the
long
running,
ci
jobs
are
waiting,
we
have
two
scenarios
that
can
happen.
Things
go
succeed.
B
Compliance
is
broken
and
a
developer
is
informed
of
this,
and
we
have
a
72-hour
clock
that
kicks
off
this
72
hours
allows
the
developer
to
be
able
to
fix
the
issue
that
was
there
on
the
github
and
a
price
server,
build
and
reverse
that
or
revert
the
change
that
they
made.
If
the
fix
or
the
change
didn't
happen
within
the
72
hours,
a
big
hammer
is
thrown
and
all
deployments
to
github.com
is
halted
until
the
quality
of
github
enterprise
server
is
restored.
B
B
Why,
while
developing
the
solution
was
to
gather
context,
collect
the
ideas
and
find
something
that
was
maintainable
so
as
we
build
deferred
compliance
workflow
in
rci
system,
we
leveraged
our
products
issues,
actions
workflow
integrations.
We
focused
on
the
key
metrics
to
identify
bottlenecks
and
found
a
maintainable
and
a
reusable
solution.
We
ultimately
built
an
internal
tool
that
supports
our
developers
and,
at
the
same
time,
acts
as
a
guard
rail
for
product
qualities
and
as
for
numbers,
we
reduce
our
ci
time
to
13
minutes,
making
our
ci
workflow
three
times
faster.
B
Then,
as
an
added
bonus,
we
reduced
our
compute
cost
down
by
a
fourth.
This
was
improvement
done
just
in
one
of
the
work
streams
in
the
larger
and
safe
in
the
larger
initiative.
There
was
a
lot
more
improvement
as
liz
talked
about,
but
we
focused
on
the
fundamentals
of
development
tracking
down
important
metrics,
deploy
frequency
change,
fail
to
rate
time
to
restore
a
service
and,
of
course,
what
I
fix
lead
time
to
change
above
and
beyond
numbers.
B
Github
engineering
work
together
to
identify
prioritize
plan
swarm
on
the
most
important
impactful
issues
that
matter
to
us.
We
brought
together
teams
from
different
parts
of
organization
to
work
on
the
fundamentals
and
what
warmed
my
heart,
the
most
is
how
we
were
able
to
focus
on
our
engineering
platform,
cracking
down
technical
debt,
making
engineering
improvements
so
that
we
could
continue
to
build
the
best
in
class
developer
products.
For
you
with
that.
Thank
you
so
much
for
listening
to
how
we
fixed
and
worked
on
technical
debt
at
github.