►
Description
Presented by Rob Zuber, CTO (CircleCI).
This session looks at anonymized team data to share insights, behaviors, and metrics that help teams build better software, faster. We'll examine the anonymized, aggregate data from millions of builds for best practices on commits, pull requests, disaster recovery, frequency of deploy, and more.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Welcome
so
I
wanted
to
start
out.
I
wanted
to
start
out
this
morning
by
taking
everybody
back
a
few
years
or
a
few
years
in
my
life,
which
is
actually
20
years
to
1998
and
my
first
startup,
but
the
story
isn't
really
about
working
in
a
start-up
because
also
in
1998.
At
the
same
time,
I
started
dating
the
woman
who
is
now
my
wife,
and
this
is
a
story
a
little
bit
about
her,
but
more
importantly,
about
Christmas,
turkey,
and
so
we've
been
dating
for
about
three
months.
A
Let's
say
and
and
I
had
the
opportunity
for
the
first
time
to
meet
her
parents.
Everybody
loves
meeting
the
parents
of
the
person
they're
dating
right,
and
so
she
invited
me
over
for
dinner,
but
she
only
gave
her
parents
about
20
or
30
minutes
of
notice,
and
it
was
a
couple
days
after
Christmas,
and
so
they
were
only
planning
on
having
leftovers
leftover,
turkey
and
when
I
arrived
at
the
house,
they
were
very
apologetic
about
this.
We're
really
sorry.
All
we
have
is
Turkey
and
I
thought.
That's
weird,
I
love
Turkey!
A
I
mean
christmas
leftovers,
one
of
my
favorite
things.
Why
are
they
so
upset
about
this?
So
so
we
sit
down
and
we're
eating
dinner
or
we're
getting
ready
to
eat
dinner
right
and
the
foods
being
passed
around
the
table
and
Dana.
That's
my
wife.
So
her
her
father
is
sitting
on
my
right
and
he
passes
me.
Some
beans
and
I
take
some
beans
and
he
passes
me.
A
Some
salad
I
take
some
salad
and
then
but
the
turkey
comes
around
and
he
he
passes
it
over
me
to
Dana's
sister
who's
sitting
on
my
left
and
I
thought.
Well,
that's
kind
of
weird,
but
I
mean
I.
I
just
met
these
people
right,
I'm,
not
gonna,
be
confrontational
about
it
is
this?
Is
this
a
rite
of
passage?
Do
you
have
to
come
for
dinner
five
times
before?
Anyone
will
give
you
the
turkey
like
like
what
exactly
is
going
on
here,
so
we
eat.
A
You
know
I
clear
my
plate
of
of
the
salad
and
beans,
a
little
hungry,
still,
I
got
to
say
and
and
so
everyone's
kind
of
looking
around
there's
a
there's,
a
weird
awkward
silence
and
I'm
people
are
looking
at
me
because
they
can
see
that
I'm
eyeing
the
various
forms
of
food
at
the
table
and
and
then
finally
Dana's
mom.
She
says
Rob
can
I
get
you
anything
else,
and
now
I'm,
like
I'm,
put
on
the
spot
right
so
I
said,
can
I
like.
Would
it
be?
A
Okay
can
I
got
some
turkey
and
everyone
stops,
and
everyone
turns
and
looks
at
Dana
and
Dana
looks
at
me
and
says:
I
thought
you
were
vegetarian
and
the
thing
is
I
was
just
broke
right,
I
was
working
at
a
startup,
it
was
like
friends
and
family
funded
and
I
thought,
because
I
was
young
and
foolish.
The
way
to
win
the
heart
of
this
woman
that
I
was
dating
was
to
take
her
to
all
the
finest
restaurants
in
Toronto,
where
I
was
living
at
the
time,
but
I
couldn't
afford
it.
A
So
this
is
the
key
point
that
I
want
to
leave
with
you
here
today,
which
is
that
you
need
to
own
your
own
story,
because
this
is
happening
every
day
around
you,
it's
true
and
dating
it's
true
in
life.
It's
true
at
work,
if
you're
not
clearly
conveying
to
people
what's
happening
and
and
why
things
are
happening,
they're
making
up
their
own
impressions
in
their
mind
and
they're
deciding
what
the
story
is
and
and
telling
everybody
I
mean
she
told
her
parents
all
this.
She
told
all
her
friends.
I
was
vegetarian.
A
We
had
never
ever
talked
about
it,
bringing
us
a
little
bit
closer
to
home
into
it.
Well
closer
to
work.
Let's
say
before
that
start
up
in
my
first
job
out
of
college
I
worked
in
a
multinational
manufacturing
facility,
so
as
a
factory
and
I
was
a
process
engineer
and
I
was
responsible
for
the
quality
and
effectiveness
of
the
process.
As
we
delivered,
we
were,
it
was
think
Foxconn
we're
building.
A
You
know
electronic
circuit
boards,
basically
and
after
my
first
year,
I
had
a
horrific
annual
review,
I'm
kind
of
proud
of
it
actually
I'm,
not
proud
to
say,
I'm
super
proud
of
it,
and
the
feedback
that
I
got
from
my
manager
was
that
she
had
been
up
on
the
factory
floor
over
the
course
of
the
year
and
had
heard
from
folks
that
worked
up
there.
That
I
was
slowly
disengaging
over
the
course
of
that
year.
A
It
was
so
obvious
to
me
the
value
that
I
was
delivering,
that
I
had
never
bothered
to
talk
about
it
with
my
manager.
I
just
talked
to
the
other
engineers,
and
we
figured
out.
This
was
the
thing
we
had
to
do
and
I
spent
all
of
my
time
doing
it
so
again,
very
very
important
to
be
telling
that
story
and
talking
about
the
value
that
were
that
we're
delivering
so
we're
software
engineers
right
most
of
us,
I'm
assuming
or
in
software
engineering
management,
something
like
it.
How
do
we
measure
value?
A
How
do
we
measure
the
value
that
we're
delivering
as
software
engineers
we're
not
very
good
at
it?
I'll
be
honest
with
you
and
I'll
tell
you.
We
haven't
been
very
good
at
it
for
a
very,
very
long
time.
In
fact,
I
would
say:
we've
been
bad
at
it
for
a
very,
very
long
time.
I
don't
know.
If
anybody
remembers
this
little
gem,
hopefully
you
can
see
it
up
there,
but
I
downloaded
this
just
for
fun.
A
It's
a
source
line
of
code,
counter
that
is
copy
right
through
2004,
so
I'm,
assuming
we've
given
up
on
this,
but
I
ran
it
across
the
project
that
I
wrote
to
generate
the
charts
for
this
talk,
576
lines
of
code,
not
a
lot,
but
take
a
look
at
the
numbers.
Apparently
that
should
have
taken
me
two
point,
eight
months
and
that
would
be
15,000
dollar
value
where
value
is
measured
in
what
you
pay
software
engineers.
If
you
look
to
the
left,
you
can
see
the
average
salary
of
a
software
developer.
A
Clearly,
there's
some
updates
to
this
to
this
to
be
done.
But
what
does
that
mean?
It
doesn't
really
mean
anything.
First
of
all
that
was
about
four
hours
of
work,
not
2.8
months,
so
the
calculation
is
clearly
wrong
and
how
did
I
deliver
so
much
value
in
four
hours
copy
and
paste?
My
friends,
this
is
not
important
to
me.
This
is
not.
This
is
not
a
sustainable.
Well
tested!
Well
developed
easy
to
maintain
set
of
code,
it's
a
bunch
of
stuff
that
I
threw
together
to
make
some
charts
and
then
I'm
done
with
it
right.
A
So
how
am
I
thinking
about
that
value?
Is
that
really
what
we
should
be
thinking
about?
No
so,
as
I
said,
we've
given
up
on
this,
but
we
haven't
found
a
good
replacement.
We've
basically
concluded
that
you're
effective
if
you're
very
busy,
but
busy
doing
what
busy
digging
through
a
labyrinth
of
tech
debt
trying
to
get
something
out
the
door
busy
fighting
your
flaky
tests
in
your
build
environment,
trying
to
actually
get
the
thing
that
you
built
out
into
production,
busy
arguing
over
white
space
in
your
PRS.
None
of
those
things
really
feel
valuable
right.
A
So
when
we're
cranking
it
out
all
hours
of
the
night
and
chalking
up
all
the
Red
Bull
cans
on
our
desk,
like
hey
I'm,
I'm,
really
crushing
it
when
you
ship
that
thing.
Finally,
at
4:00
a.m.
and
you're
super
proud,
I
think
the
question
is:
why
wasn't
able
to
do
it
at
4:00
p.m.
right
like
why
couldn't
I
have
got
this
done
in
a
normal
amount
of
time
and
go
home
busy?
Does
not
equal
effective
as
a
commutable
equation?
A
This
is
how
I
want
to
feel
when
I
work,
partly
because
I
got
something
done
and
now
I
can
actually
go
skiing
and
do
something
fun
that
I
want
to
do
with
my
life,
but
also
more
metaphorically,
this
weightless
feeling,
if
I'm
not
being
dragged
down
by
all
the
things
around
me,
they're
slowing
me
down
and
keeping
me
from
delivering
what
I
really
want
to
do.
Okay,
let's
be
honest
as
engineers:
do
we
want
to
do
battle
with
our
systems?
No,
we
want
to
deliver
value
right.
A
We
want
to
deliver
things
that
our
customers
are
excited
about,
that
are
building
on
the
business.
That
are
the
interesting
problems,
I
mean
even
if
it's
not
the
sort
of
UI
customer
facing
stuff.
We
want
to
solve
the
interesting
problems
and
we're
solving
the
kind
of
mundane
problems.
I
mean:
let's
talk
about
flaky
tests,
would
you
go
off
this
ramp
if
it
kind
of
work,
sometimes?
And
but
you
never
really
knew
I,
don't
think
so
right.
This
is
trust
in
your
system
and
knowing
that
things
are
gonna
work
effectively.
A
That's
what
gets
you
this
this
feeling
of
flight
we'll
call
it
so
I'm,
the
CTO
of
circle,
C
I,
think
you
figured
that
out
from
my
cover
slide,
but
but
one
of
the
things
that's
really
awesome
about
working
at
circle.
Ci
is
where
we
happen
to
sit
and
what
what
I
mean
by
that
I
mean
our
offices
are
lovely,
but
but
where
we
sit
in
the
developer
pipeline,
so
we
are
software
developers.
I
am
a
software.
A
Developer
have
been
for
a
very
long
time,
I
run
a
team
of
software
developers
and
we
think
about
software
development
for
our
own
sake.
We
want
to
be
good
at
what
we
do,
but
we
also
have
the
opportunity
from
25,000
plus
organizations
to
look
at
who
else
is
out
there
and
how
they
act
and
what
they
do,
and
this
has
given
us
an
opportunity
to
really
sit
down
and
think
about
this
problem.
What
does
it
mean
to
be
effective?
What
does
it
mean
to
deliver
value
and
so
I'm?
A
Actually,
not
a
software
engineer
by
training
I'm,
a
I
studied
engineering
physics,
so
I
have
a
little
science
in
my
my
background,
I
think
about
this
and
sort
of
the
terms
of
the
scientific
method
right,
let's
figure
out
a
hypothesis.
Let's
go
test
that
hypothesis
and
then
let's
draw
some
conclusions
or
more
likely,
find
out
we're
wrong
and
start
with
a
new
hypothesis
and
go
through
the
process
a
few
times,
but
we
only
have
40
minutes
so
we'll
we'll
streamline
the
process
a
little
bit.
A
We
went
looking
for
some
clues
and
one
of
the
interesting
things
that
we
discovered
pretty
early
just
sitting
down
and
and
not
digging
into
any
data
or
doing
any
real
analysis,
but
just
talking
through
what
have
we
heard
from
customers,
one
of
the
things
that
I've
heard
so
I've
been
at
circle
CI
a
little
over
four
years,
and
one
of
the
things
that
I
hear
consistently
and
have
heard
for
four
years
is
my
build
time
is
critical.
If
my
build
time
crosses
X
threshold,
then
my
engineers
are
wasting
their
time.
They
can't
be
productive.
A
A
A
So,
let's
talk
about
builds,
we
run
a
lot
of
them.
We
see
a
lot
of
them
and
I,
don't
know
how
many
of
you
are
circle,
CI
customers
or
use
CI
at
all.
But
if
you're
like
me,
the
green
build
is
a
little
bit
exciting
like
this
is
progress
right,
so
what
does
it
mean
to
be
a
green,
build?
First
of
all,
let's
focus
on
master
right.
A
If
you're,
a
github
user,
you
probably
think
of
the
primary
or,
if
your
git
user
I,
guess
you
probably
think
of
your
primary
or
default
branches
master,
there
might
be
another
name
for
it:
I'm
gonna
call
it
master
for
the
sake
of
this.
Every
other
branch
is
a
little
bit
of
noise
right.
It
might
just
be
safety.
Commit
I
had
this
on
my
laptop
I
needed
it
to
be
somewhere
else,
so
I
committed
it
to
to
github
that
doesn't
really
matter
right,
but
the
master
branch
is
precious.
A
This
is
where
all
the
work
is
coming
together
and
getting
out
through
the
pipeline
and
into
production,
or
maybe
just
into
your.
You
know
your
docker
registry
or
whatever
the
asset
might
be,
and
so
every
single
time
you
get
a
green,
build
you're
a
hero.
This
is
exciting,
we're
winning
we're
making
progress.
Maybe
we're
defeating
the
Empire
I,
don't
know
that
might
be
a
little
bit
hard
to
go
or
far
to
go
on
a
green
bill,
but
it's
exciting
right.
A
It's
it's
like
a
little
boost
of
excitement
and
enthusiasm
and
energy
that
comes
from
this
achievement
right.
What
about
red
builds
red
builds
are
terrible,
think
about
master
branch.
Red
builds
right.
How
does
that
even
happen?
You,
you
probably
built
on
your
branch
and
then
you
merged
it
and
someone
else
built
on
their
branch
and
they
merged
it,
and
it
failed
now
like
in
a
lean
culture,
someone's
yanking,
the
andon
cord
right,
stop
everything.
Let's
go
fix
the
build.
How
did
we
get
into
this
situation?
A
Flaky
tests
anybody
have
flaky
tests.
Anyone
ever
seen
a
flaky
test,
maybe
one
or
two
yeah.
As
a
software
development,
community
I
can
tell
you
again
from
where
I
sit,
we're
actually
pretty
bad
at
writing,
consistently
good
tests
and
this
burns
a
ton
of
time.
Think
about
that,
like
the
test
failed,
how
many
times
you
have
the
conversation?
Oh
I
know
that's
a
flaky
test.
Don't
worry
about
it!
We'll
just
run
it
again
or
even
better,
we'll
just
push
the
production,
because
we
don't
have
time
to
run
the
test.
A
Speed
again
and
I
know,
that's
flaky!
So
don't
worry
about
it
right,
okay,
so
that's
one
option.
The
other
is
software
complexity.
If
your
software
is
so
complex
that
two
people
merging
means
all
bets
are
off
in
terms
of
whether
or
not
you're
actually
gonna
get
a
good
result,
then
you
should
probably
spend
some
time
on
that
right.
A
This
is
how
I
feel
about
red
builds.
Red
robes
are
terrible,
they're
terrible,
and
so
we
want
to
avoid
them.
So
as
we
sat
down
and
thought
through
this,
that
was
the
first
thing
that
really
came
to
mind.
Was
master
branch
stability
like
how
good
are
we
at
keeping
the
master
bench
in
a
good,
stable,
State
right
number
one,
and
with
that
we
talked
about
red,
builds
and
just
the
frustration
of
red
builds
and
and
slowing
people
down,
and
no
one
else
can
move.
We
also
realize
that
there's
time
spent
there.
A
So
if
you're,
a
good
organization
and
you're
builds,
go
red,
everybody
stops
what
they're
doing
they
figure
it
out.
They
get
it
green
and
you
move
on
some
organizations,
maybe
are
like
I.
Don't
know
that
seems
frustrating,
let's
go
home
for
the
weekend
and
we'll
take
a
look
at
it
on
Monday
and
no
one
can
get
anything
done
over
that
period
right
and
then
we
start
to
look
at
other
things,
and
this
is
about
a
year
ago
we
were
thinking
through
just
again.
A
What
can
we
do
without
diving
into
our
data,
and
we
looked
at
just
through
some
conversations
with
folks
deploy
frequency
like
just
how
fast
are
you
getting
things
out
and
deploy
time?
How
long
does
it
take
you
to
get
something
out
into
production
at
the
same
time
or
around
the
same
time
in
terms
of
research
but
published
more
recently
Nicole
for
his
grandes,
humble
and
Jean
Kim
put
out
this
book
accelerate
it's
based
on
their
DevOps
reports,
which
you
maybe
maybe
haven't,
had
a
chance
to
read,
and
they
came
to
some
pretty
similar
conclusions.
A
If
you
look
at
how
they
did
their
research,
it
involved
a
lot
of
surveys
and
it's
got
a
lot
of
scientific
depth
to
it,
which
is
super
awesome
and
I'm.
Glad
that
they're
doing
this
work,
but
we
looked
at
that
and
said
well
what
if
we
could
actually
just
pull
that
out
of
our
data
and
then
we
could
correlate
against
all
the
other
things
all
the
other
patterns
that
we
see
people
doing
versus
having
to
go
back
and
sort
of
do
more
surveying
and
digging
into
these
organizations.
A
So
we
came
up
with
a
formula
and
now
I
will
tell
you
this
is
this
is
scratching
the
surface,
we're
still
figuring
this
out,
but
we
tried
to
look
at
these
factors
that
we've
seen
and
figure
out.
Okay,
what
really
matters
and
again
going
back
to
what
we
see
every
day
as
builds
flow
through
our
system
and
what
we
recognize
as
key
factors
in
understanding
your
ability
to
get
things
done
and
deliver
value.
We
sort
of
sat
down
and
came
up
with
this
formula.
A
There's
a
second
key
point
with
that,
which
is
we
could
have
built
a
neural
network
or
some
other
ml
thing
that
I
totally
don't
understand
and
would
just
farm
out
to
you
know
google's
AI
platform
or
whatever.
But
then
what
do
you
get
out
of
that?
You
get
back
a
number
and
you
don't
know
what
to
do
with
it.
You
have
no
idea
how
it
was
derived.
You
have
no
idea
what
the
key
factors
are,
whereas
I
think
when
we
talk
of
builds,
people
really
inherently
understand
that.
A
So
this
is
the
approach
we
took.
I'm
gonna
walk
through
a
little
bit
of
math,
it's
not
super
complicated.
That
was
one
of
our
goals.
But
basically,
if
you
look
at
the
numerator,
you
start
out
at
a
top
line
of
1
and
then
effectively
get
penalized
for
having
a
bad
ratio
of
red
builds
to
green
builds
and
at
the
point
that
your
red
builds
out
number
your
green
builds
you
head
into
the
negatives,
so
we
basically
I
didn't
put
the
limits
on
here.
We
we
cap
at
minus
1
cuz
beyond
minus
1.
A
It's
not
really
that
interesting
to
worry
about,
but
we
basically
have
this
range
minus
1
to
1
as
a
simple
score
to
say
this
is
a
project
that's
working
effectively
and
then
in
the
denominator.
We
want
to
reduce
the
magnitude
of
the
direction
so
again
thinking
about
the
time
that's
been
spent.
If
you
do
have
red
builds,
but
you
can
get
them
resolved
quickly
and
you
generally
keep
your
stuff
in
a
green
state.
A
That's
a
sign
that
you're
moving
effectively
because
you're
not
blocking
everybody
else
on
the
team,
it's
a
little
bit
weird
that
it
reduces
the
magnitude
and
the
negative,
but
honestly
once
you're
in
the
negatives,
once
you
have
more
red
builds
than
green
builds
on
your
master
branch.
The
magnitude
probably
doesn't
super
matter.
You
should
definitely
put
some
work
into
cleaning
that
up.
A
Ok,
so
we
came
up
with
this
hypothesis
of
maybe
this
is
an
interesting
thing
will
tell
us
something
about
some
projects.
What
happens
if
we
go
look
at
our
data
and
try
to
refine
it,
so
we
had
three
questions.
What
fits
what
looks
into
it
even
makes
sense
to
us
what
doesn't
what
kind
of
stands
out
and
what
else
can
we
learn?
A
A
So
number
one
again,
our
again
I
didn't
say
this
before
we're:
not
gonna
I'm
not
going
to
claim
causation.
But
if
you
look
at
this
distribution,
which
is
basically
total
builds-
and
this
is
all
over
the
period
of
one
month,
which
was
August
of
this
year
by
projects
and
look
at
the
nvs
distribution.
What
you
don't
see
is
projects
that
we
would
consider
to
be
negative
projects
that
are
not
good.
A
Moving.
A
lot
of
builds,
it's
difficult
to
be
effective
and
deliver
at
a
high
rate.
If
you
have
lots
of
red,
builds
again
everybody's
stopping
what
they're
doing
and
focusing
on
fixing
things,
maybe
they're
or
maybe
they're
just
sitting
there
bleary-eyed
at
3
o'clock
in
the
morning
hitting
rebuild,
rebuild,
rebuild
hoping
to
get
past
those
flaky
tests
instead
of
getting
something
new
done.
A
So
that's
interesting
makes
sense.
We
sort
of
say
ok
that
validates
sort
of
what
we're
thinking
about
here
and
then
we
took
a
different
look,
which
is
what,
if
we
try
to
think
about
units
of
value
in
some
other
way,
and
if
you
follow
a
git
flow,
you
probably
have
topic
branches
and
if
you
have
topic
branches,
they're,
probably
related
to
topics
like
this
unit
of
you
know:
functionality
that
I'm
trying
to
deliver
this
bug
that
I'm
trying
to
fix.
A
And
so,
if
we're
churning
through
a
lot
of
a
lot
of
branches,
then
that's
a
sign
that
we're
moving
the
needle
it's
a
theory,
but
it
lines
up
as
well,
which
is
you
don't
see
in
the
bottom
right
here?
People
moving
a
lot
of
value
and
doing
it
in
a
way
where
their
build
is
very
unstable
and
inconsistent.
A
Ok,
so
what
can
we
learn?
What
about
team
size,
I
love!
This
story
that,
like
to
pizza
team,
is
the
perfect
team.
But
what
does
that
found
it
on
how
easy
it
is
to
order
two
pizzas
I'm,
not
100%
sure?
So
what
we
see
in
this?
This
is
team
sizes
based
on
active
developers,
people
committing
to
the
codebase
across
projects
in
our
we're
gonna
air,
in
on
our
platform
and
the
distribution.
So
there
there
are
teams
well
beyond
25,
first
of
all,
but
it
gets
pretty
weird
out
there.
A
It's
having
200
people
working
on
a
repo
I
think
is
kind
of
an
outlier
and
probably
not
something
that
you
want
to
spend
a
lot
of
attention
on
unless
you're,
Google
or
Facebook,
and
then
it's
thousands
for
those
not
familiar
with
box
plots.
The
little
green
lines
in
the
middle
of
the
box
are
the
median
and
the
box
spans
the
sort
of
top
of
the
first
to
top
of
the
third
quartile,
and
then
the
whiskers
are
a
prediction
of
the
full
scope.
A
What
I
took
off
of
this
chart
to
make
it
easier
to
read
was
the
outliers
and
if
you
look
at
team
sizes,
one
through
five,
you
basically
span
all
the
way
to
negative
one
and
all
the
way
to
one.
What's
interesting
about
that,
I
mean
if
you
have
a
team
size
of
one
working
on
a
project.
It's
probably
not
really
important
right.
It's
someone's
little
side
project
they're,
just
maintaining
this
thing
to
do
their.
You
know
notes
whatever.
A
Although
it's
interesting
to
me,
that
people
would
put
their
notes
in
CI
and
then
there's
kind
of
a
point
in
the
middle
there
around
seven
eight
nine,
two
pizzas,
where
the
variability
gets
much
smaller
right
and
so
the
teams
that
the
median
is
kind
of
consistent,
but
the
the
likelihood
that
you're
gonna
be
way
out
of
that
range
gets
a
lot
smaller
and
then,
once
you
cross
past
that
it
gets
all
over
the
place.
I
mean
look
at
twenty
to
twenty
five.
It's
super
squirrely.
A
So
organizational
culture.
This
is
a
really
interesting
one.
To
me,
this
is
twelve
organizations
who
are
at
the
large
or
the
high
end
of
number
of
projects
on
our
platform.
They're
not
actually
named
0.301,
but
this
is
all
anonymized
data.
So
the
number
that's
in
there
as
the
name
of
the
plot,
is
they're
weighted
or
aggregate
nvs
across
the
organization,
and
these
are
histograms
from
negative
one
to
one
based
on
project.
A
So
if
you
look
at
the
top
left-
and
in
fact,
if
you
look
everywhere,
everybody
has
a
crappy
project-
everybody
has
a
project
that
they,
maybe
they
just
don't
care
about
it.
But
again,
why
is
why
are
they
running
CI
and
CD
on
a
project?
That's
not
important!
That's
an
interesting
question,
but
if
you
look
to
the
bottom
right,
you
know
where
you
have
a
high
NB
s.
A
Overall,
you've
got
an
organization
that
skews
all
of
their
work
towards
paying
a
lot
of
attention
to
the
quality
of
their
builds,
and
if
you
look
to
the
top
left,
you've
got
an
organization
that
still
has
some
good
stuff.
I
mean
everybody
has
stuff
up
at
the
top
end
as
well,
but
they
don't
care
as
much
about
all
of
their
projects.
So
there's
some
interesting
digging
to
do
here
is
it?
Is
it
about
teams,
individual
teams?
A
Is
it
about
specific
projects
in
the
organization,
not
100%
clear,
but
this
gives
us
something
to
go
talk
to
people
about,
and
why
would
you
ever
have
a
conversation
about
code
quality
without
fighting
about
languages?
That
would
be
crazy,
so
I
love
this
chart
because
we're
a
closure
shop
and
I
a
hundred
percent
swear
I
did
not
fix
this
chart.
I
did
not
take
the
closure
number
and
just
move
it
up
to
the
top,
but
there
are
a
couple
of
intuitively
interesting
things
in
here,
like
the
dynamic
languages
being
kind
of
clustered
together
right.
A
The
way
that
people
write,
Python
and
Ruby
I,
don't
know
about
shell,
we'll
just
assume,
there's
no
tests
on
those
those
kind
of
cluster
together
in
JavaScript
right
as
you
get
down
below
you
kind
of
move
into
type
languages.
Does
that
tell
us
something
specific
I
doubt
it
I'm,
not
gonna,
stand
up
here
on
stage
and
tell
you
to
stop
using
type
languages
and
move
to
closure
I'm.
A
Just
not,
but
this
is
interesting
stuff
to
think
about
and
then
at
the
very
bottom
you've
got
Objective
C,
C++
and
Swift
I
mean
Swift
I,
don't
know
what
to
say
about
that,
but
mobile
development
is
sort
of
notoriously
hard
right.
Like
it's
hard
to
test
it
effectively.
You
could
see
scenarios
where
the
code
is
complex
and
getting
a
good,
stable,
build
process
will
be
hard,
so
it
lines
up
it
kind
of
makes
sense,
and
it's
interesting
to
think
about.
A
Okay.
So
let's
go
talk
to
some
customers
and
find
out
what
they
say
right
and
the
thing
that
we
kept
hearing
when
we
talk
to
customers
was
oh
yeah,
that's
because
so
let's
talk
about
this
customer.
This
is
the
first
chart
that
I
showed,
which
is
just
total,
builds
against
MVS,
and
we
know
this
customer.
We
went
talked
to
them
and
we
said
hey.
You
know
you
have
this
bill,
that's
it's
actually
moving
pretty
rapidly
this
project.
A
lot
of
work
is
getting
done,
but
it's
like
constantly
broken.
A
What
are
you
doing
and
they
said
oh
yeah
yeah?
That's
because
we
have
this
this
linter
that
basically
never
works
okay.
So
what
we
do
is
we
ignore
that
job
in
the
work
flow?
So
this
is
a
work
flow
and
they
just
have
a
job
to
nowhere
that
they
expect
to
fail
and
they
ignore
it.
But
the
result
of
the
work
flow
is
red
and
the
way
they
determine
whether
they
should
actually
keep
going.
A
Is
they
go
and
look
at
the
output
check
all
the
jobs
and
make
sure
that
the
job
they
expect
to
fail
failed
that
that's
the
only
reason
that
it's
failing
not
super
productive
I
mean
there's
a
reason
that
we
post
your
status
back
to
github.
So
you
can
just
look
at
it
and
go
okay,
I'm
good.
Not
so
you
can
click
through
and
go
read
the
output
and
say:
oh
okay,
yeah
the
thing
that
I
expected
to
fail
failed.
Everything
else
is
fine.
A
Let's
ship
to
production
you'll
be
unsurprised
to
learn
that
this
customer
is
basically
exactly
the
same.
This
is
a
super
high
volume
customer
doing
tons
of
work,
massive
team
and
a
slightly
different
scenario,
but
they
have
a
job
that
requires
on
a
third
party
server.
That
requires
or
depends
on
a
third
party
service
and
and
their
free
trial
ran
out,
so
they
just
kept
building
and
they
just
ignore
it
in
every
time
the
workflow
fails
they
go
and
look
at
the
workflow
and
they
go
yeah
the
thing
with
the
free
trial.
A
That's
the
thing
that
failed.
Everything
else
is
fine.
We
can
ship
now
I
mean.
This
is
why
you
have
this
data.
If
I
had
this
data
and
I
was
an
engineer,
I'd
go
to
my
boss
and
say
look,
this
is
a
problem.
We
are
not
able
to
get
more
work
done
because
you
won't
pay
for
this
$99
a
month
trials
or
thing
that
the
trial
has
expired
and
have
the
conversation
at
least
either
it's
useless.
So,
let's
take
it
out
of
our
build
or
it's
worth
$99.
A
A
Ok,
so
what
does
this
all
do
for
you?
What
is
the
opportunity
here
for
all
of
us,
the
opportunity
of
owning
our
own
story,
so
I
think
notoriously
as
engineers
we
are
reluctant
to
be
measured
every
time
someone
shows
up
with
some
measurements.
Our
response
is
you're.
Measuring
me,
I!
Don't
want
that
right.
I,
don't
want
to
be
graded
on
my
ability
to
move
three
story.
Points
versus
two
four
story
points
across
the
board
this
week.
This
is
about
me
and
I
feel
uncomfortable,
but
as
a
result,
we
create
this
void
right.
A
We
create
the
void
and
the
story
and
the
void
gets
filled
in
trust.
Me
I
can
tell
you
as
a
CTO
as
a
manager
when
I
was
a
manager
as
an
engineer
as
a
human
in
society,
the
void
gets
filled
in
so
you
want
to
do
the
filling
that
feeling
of
flight.
This
is
what
you
want.
You
want
to
be
released
right.
You
want
to
remove
all
the
weight,
that's
on
you
and
be
able
to
focus
on
things
that
matter.
A
But,
let's
be
honest,
this
is
a
canary
metric.
Sorry,
if
this
is
disturbing,
we
talk
about
Canaries.
All
the
time-
and
it
occurred
to
me
that
actually
they
were
Canaries
like
real
Canaries.
That's
got
a
freaky,
but
don't
worry
they
stopped
now.
They
have
little
digital
meters
to
check
the
carbon
monoxide
level,
but
it's
a
canary
metric
right.
It's
not
gonna
tell
you.
This
is
the
thing
that's
wrong
in
your
process,
but
it
is
gonna.
Tell
you
your
project
relative
to
everybody
else's
projects.
It's
not
that
good.
A
Maybe
you
should
invest
some
time
so
that
you
can
move
a
little
more
effectively
as
a
team.
So
let's
say
this
is
you-
and
this
is
your
feeling
about
the
flaky
tests
in
your
environment
right
every
day?
Oh,
my
god.
Here's
the
flaky
test
again
sure
everybody
go
to
lunch.
I'll
just
keep
hitting
the
retry
button.
Maybe
by
the
time
you
get
back,
we'll
have
a
green
bill
that
we'll
be
able
to
put
this
thing
in
production.
A
Don't
live
like
that
right
prove
to
the
people
that
you
work
with
the
people
that
you
work
for,
and
this
is
not
how
everyone
works.
Let's
share
and
open
the
conversation
and
be
able
to
say,
hey,
I
need
to
get
some
time
to
fix
these
crazy
flaky
tests.
I
mean
I
as
a
CTO
now
fix
flaky
tests,
because
I'm
like
this
is
crazy.
What
are
we
doing?
Yeah
you're
welcome?
A
So
what
should
you
take
away
from
this
now?
Invest
in
your
effectiveness,
if
you're,
not
it's
slowing
you
down,
and
somebody
else
out
there,
who's
working
on
the
same
business
as
you
is,
is
investing
in
their
effectiveness
and
they're,
making
faster
progress
against
the
things
that
really
matter:
they're,
delivering
value
and
and
they're
gonna
win
number
to
own
your
story.
A
The
story
is
getting
filled
in
people
are
looking
at
what's
happening
and
if
they
don't
have
the
information
that
you
have
they're
gonna
make
their
own
decisions
their
own
ideas
about
what's
happening,
and
it's
probably
gonna
drive
you
nuts
and
then
number
three
share
the
more.
We
can
talk
about
the
things
that
really
aren't
secret
or
different
about
us
as
engineering
teams,
the
more
we
can
all
focus
on
the
things
we
really
want
to
focus
on,
which
are
the
interesting
problems,
delivering
value
and
shipping
at
4
p.m.
instead
of
4
a.m.
A
A
Let's
talk
about
it,
I
mean
I'm
I'm
interested
in
hearing
from
you.
If
you
are
interested
in
this,
if
you
want
to
know
what
this
means
for
you,
if
you
have
ideas
about
the
kinds
of
work
that
you
don't
get
to
do,
that
you
could
focus
on
if
they
were
supported
with
data,
I
would
love
to
hear
about
them
because
we're
very
interested
again
as
an
organization,
not
just
as
software
developers
but
people
sitting
in
the
development
pipeline
to
think
about
what
it's
going
to
take
to
make
everybody
around
us
more
effective.